2023-02-13 17:00:22,341 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa 2023-02-13 17:00:22,354 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-13 17:00:22,370 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 17:00:22,370 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4, deleteOnExit=true 2023-02-13 17:00:22,371 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 17:00:22,371 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/test.cache.data in system properties and HBase conf 2023-02-13 17:00:22,372 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 17:00:22,373 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/hadoop.log.dir in system properties and HBase conf 2023-02-13 17:00:22,373 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 17:00:22,374 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 17:00:22,374 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 17:00:22,489 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-13 17:00:22,891 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 17:00:22,895 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 17:00:22,895 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 17:00:22,895 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 17:00:22,896 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 17:00:22,896 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 17:00:22,896 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 17:00:22,896 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 17:00:22,897 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 17:00:22,897 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 17:00:22,897 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/nfs.dump.dir in system properties and HBase conf 2023-02-13 17:00:22,897 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/java.io.tmpdir in system properties and HBase conf 2023-02-13 17:00:22,898 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 17:00:22,898 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 17:00:22,898 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 17:00:23,366 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 17:00:23,370 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 17:00:23,643 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-13 17:00:23,799 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-13 17:00:23,814 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:00:23,851 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:00:23,883 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/java.io.tmpdir/Jetty_localhost_40913_hdfs____.vpni9j/webapp 2023-02-13 17:00:24,014 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40913 2023-02-13 17:00:24,050 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 17:00:24,051 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 17:00:24,425 WARN [Listener at localhost/45307] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:00:24,523 WARN [Listener at localhost/45307] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 17:00:24,541 WARN [Listener at localhost/45307] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:00:24,546 INFO [Listener at localhost/45307] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:00:24,551 INFO [Listener at localhost/45307] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/java.io.tmpdir/Jetty_localhost_37333_datanode____.x6s5uy/webapp 2023-02-13 17:00:24,644 INFO [Listener at localhost/45307] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37333 2023-02-13 17:00:24,941 WARN [Listener at localhost/33497] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:00:24,951 WARN [Listener at localhost/33497] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 17:00:24,956 WARN [Listener at localhost/33497] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:00:24,958 INFO [Listener at localhost/33497] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:00:24,963 INFO [Listener at localhost/33497] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/java.io.tmpdir/Jetty_localhost_34081_datanode____rb17nz/webapp 2023-02-13 17:00:25,061 INFO [Listener at localhost/33497] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34081 2023-02-13 17:00:25,074 WARN [Listener at localhost/45639] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:00:25,091 WARN [Listener at localhost/45639] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 17:00:25,094 WARN [Listener at localhost/45639] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:00:25,095 INFO [Listener at localhost/45639] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:00:25,103 INFO [Listener at localhost/45639] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/java.io.tmpdir/Jetty_localhost_38775_datanode____ey4sl3/webapp 2023-02-13 17:00:25,218 INFO [Listener at localhost/45639] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38775 2023-02-13 17:00:25,229 WARN [Listener at localhost/40167] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:00:25,419 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x33911f3d7bda133: Processing first storage report for DS-a9bef184-a9e0-4522-8f8b-4566832587af from datanode 47b434d1-9a5d-47bc-ac6d-6e8ad92f2cf9 2023-02-13 17:00:25,420 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x33911f3d7bda133: from storage DS-a9bef184-a9e0-4522-8f8b-4566832587af node DatanodeRegistration(127.0.0.1:35743, datanodeUuid=47b434d1-9a5d-47bc-ac6d-6e8ad92f2cf9, infoPort=43197, infoSecurePort=0, ipcPort=45639, storageInfo=lv=-57;cid=testClusterID;nsid=1576904320;c=1676307623443), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-13 17:00:25,421 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4838484da3eba175: Processing first storage report for DS-0430e927-fcbc-4f59-aa64-7d0960932709 from datanode 2b4e38c4-5192-482e-a428-68b79fca1402 2023-02-13 17:00:25,421 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4838484da3eba175: from storage DS-0430e927-fcbc-4f59-aa64-7d0960932709 node DatanodeRegistration(127.0.0.1:38629, datanodeUuid=2b4e38c4-5192-482e-a428-68b79fca1402, infoPort=41551, infoSecurePort=0, ipcPort=33497, storageInfo=lv=-57;cid=testClusterID;nsid=1576904320;c=1676307623443), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:00:25,421 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x51796f10d17e506: Processing first storage report for DS-4e08a395-c2a8-40ed-b610-751c45b11e60 from datanode 860591af-1224-4ad4-8461-65f27be6ecda 2023-02-13 17:00:25,421 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x51796f10d17e506: from storage DS-4e08a395-c2a8-40ed-b610-751c45b11e60 node DatanodeRegistration(127.0.0.1:33863, datanodeUuid=860591af-1224-4ad4-8461-65f27be6ecda, infoPort=41139, infoSecurePort=0, ipcPort=40167, storageInfo=lv=-57;cid=testClusterID;nsid=1576904320;c=1676307623443), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:00:25,421 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x33911f3d7bda133: Processing first storage report for DS-0c466012-55d0-41ee-809e-0b91dce84c60 from datanode 47b434d1-9a5d-47bc-ac6d-6e8ad92f2cf9 2023-02-13 17:00:25,421 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x33911f3d7bda133: from storage DS-0c466012-55d0-41ee-809e-0b91dce84c60 node DatanodeRegistration(127.0.0.1:35743, datanodeUuid=47b434d1-9a5d-47bc-ac6d-6e8ad92f2cf9, infoPort=43197, infoSecurePort=0, ipcPort=45639, storageInfo=lv=-57;cid=testClusterID;nsid=1576904320;c=1676307623443), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:00:25,421 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4838484da3eba175: Processing first storage report for DS-feb6a166-8a2d-4f4e-89ac-78a48cdaf616 from datanode 2b4e38c4-5192-482e-a428-68b79fca1402 2023-02-13 17:00:25,421 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4838484da3eba175: from storage DS-feb6a166-8a2d-4f4e-89ac-78a48cdaf616 node DatanodeRegistration(127.0.0.1:38629, datanodeUuid=2b4e38c4-5192-482e-a428-68b79fca1402, infoPort=41551, infoSecurePort=0, ipcPort=33497, storageInfo=lv=-57;cid=testClusterID;nsid=1576904320;c=1676307623443), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:00:25,421 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x51796f10d17e506: Processing first storage report for DS-1ed30d7e-6b42-41b7-9c8d-c50a93dd8caf from datanode 860591af-1224-4ad4-8461-65f27be6ecda 2023-02-13 17:00:25,422 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x51796f10d17e506: from storage DS-1ed30d7e-6b42-41b7-9c8d-c50a93dd8caf node DatanodeRegistration(127.0.0.1:33863, datanodeUuid=860591af-1224-4ad4-8461-65f27be6ecda, infoPort=41139, infoSecurePort=0, ipcPort=40167, storageInfo=lv=-57;cid=testClusterID;nsid=1576904320;c=1676307623443), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:00:25,617 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa 2023-02-13 17:00:25,678 INFO [Listener at localhost/40167] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/zookeeper_0, clientPort=54083, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 17:00:25,690 INFO [Listener at localhost/40167] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54083 2023-02-13 17:00:25,701 INFO [Listener at localhost/40167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:25,702 INFO [Listener at localhost/40167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:26,381 INFO [Listener at localhost/40167] util.FSUtils(479): Created version file at hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5 with version=8 2023-02-13 17:00:26,381 INFO [Listener at localhost/40167] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/hbase-staging 2023-02-13 17:00:26,678 INFO [Listener at localhost/40167] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 17:00:27,140 INFO [Listener at localhost/40167] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-13 17:00:27,171 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,172 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,173 INFO [Listener at localhost/40167] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:00:27,173 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,173 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:00:27,312 INFO [Listener at localhost/40167] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:00:27,383 DEBUG [Listener at localhost/40167] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-13 17:00:27,478 INFO [Listener at localhost/40167] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40255 2023-02-13 17:00:27,489 INFO [Listener at localhost/40167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:27,491 INFO [Listener at localhost/40167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:27,511 INFO [Listener at localhost/40167] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40255 connecting to ZooKeeper ensemble=127.0.0.1:54083 2023-02-13 17:00:27,552 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:402550x0, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:00:27,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:40255-0x1004ea060850000 connected 2023-02-13 17:00:27,581 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:00:27,581 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:00:27,585 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:00:27,593 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40255 2023-02-13 17:00:27,594 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40255 2023-02-13 17:00:27,594 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40255 2023-02-13 17:00:27,595 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40255 2023-02-13 17:00:27,595 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40255 2023-02-13 17:00:27,601 INFO [Listener at localhost/40167] master.HMaster(439): hbase.rootdir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5, hbase.cluster.distributed=false 2023-02-13 17:00:27,671 INFO [Listener at localhost/40167] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-13 17:00:27,671 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,671 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,671 INFO [Listener at localhost/40167] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:00:27,672 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,672 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:00:27,676 INFO [Listener at localhost/40167] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:00:27,680 INFO [Listener at localhost/40167] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43327 2023-02-13 17:00:27,682 INFO [Listener at localhost/40167] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:00:27,688 INFO [Listener at localhost/40167] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 17:00:27,689 INFO [Listener at localhost/40167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:27,691 INFO [Listener at localhost/40167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:27,693 INFO [Listener at localhost/40167] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43327 connecting to ZooKeeper ensemble=127.0.0.1:54083 2023-02-13 17:00:27,697 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:433270x0, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:00:27,699 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): regionserver:433270x0, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:00:27,700 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): regionserver:433270x0, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:00:27,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:43327-0x1004ea060850001 connected 2023-02-13 17:00:27,702 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:00:27,702 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43327 2023-02-13 17:00:27,702 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43327 2023-02-13 17:00:27,703 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43327 2023-02-13 17:00:27,703 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43327 2023-02-13 17:00:27,703 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43327 2023-02-13 17:00:27,716 INFO [Listener at localhost/40167] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-13 17:00:27,716 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,717 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,717 INFO [Listener at localhost/40167] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:00:27,717 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,717 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:00:27,718 INFO [Listener at localhost/40167] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:00:27,719 INFO [Listener at localhost/40167] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43065 2023-02-13 17:00:27,720 INFO [Listener at localhost/40167] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:00:27,720 INFO [Listener at localhost/40167] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 17:00:27,721 INFO [Listener at localhost/40167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:27,724 INFO [Listener at localhost/40167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:27,727 INFO [Listener at localhost/40167] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43065 connecting to ZooKeeper ensemble=127.0.0.1:54083 2023-02-13 17:00:27,730 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:430650x0, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:00:27,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:43065-0x1004ea060850002 connected 2023-02-13 17:00:27,732 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:00:27,732 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:00:27,733 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:00:27,734 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43065 2023-02-13 17:00:27,734 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43065 2023-02-13 17:00:27,734 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43065 2023-02-13 17:00:27,735 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43065 2023-02-13 17:00:27,735 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43065 2023-02-13 17:00:27,748 INFO [Listener at localhost/40167] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-13 17:00:27,748 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,748 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,748 INFO [Listener at localhost/40167] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:00:27,748 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:00:27,748 INFO [Listener at localhost/40167] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:00:27,748 INFO [Listener at localhost/40167] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:00:27,750 INFO [Listener at localhost/40167] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34035 2023-02-13 17:00:27,750 INFO [Listener at localhost/40167] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:00:27,751 INFO [Listener at localhost/40167] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 17:00:27,752 INFO [Listener at localhost/40167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:27,753 INFO [Listener at localhost/40167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:27,755 INFO [Listener at localhost/40167] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34035 connecting to ZooKeeper ensemble=127.0.0.1:54083 2023-02-13 17:00:27,758 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:340350x0, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:00:27,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34035-0x1004ea060850003 connected 2023-02-13 17:00:27,759 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:00:27,760 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:00:27,761 DEBUG [Listener at localhost/40167] zookeeper.ZKUtil(164): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:00:27,761 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34035 2023-02-13 17:00:27,761 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34035 2023-02-13 17:00:27,762 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34035 2023-02-13 17:00:27,762 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34035 2023-02-13 17:00:27,762 DEBUG [Listener at localhost/40167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34035 2023-02-13 17:00:27,764 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,40255,1676307626519 2023-02-13 17:00:27,773 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 17:00:27,774 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,40255,1676307626519 2023-02-13 17:00:27,793 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:00:27,793 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:00:27,793 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:00:27,793 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:00:27,793 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:00:27,795 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:00:27,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,40255,1676307626519 from backup master directory 2023-02-13 17:00:27,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:00:27,800 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,40255,1676307626519 2023-02-13 17:00:27,800 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 17:00:27,801 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:00:27,801 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,40255,1676307626519 2023-02-13 17:00:27,805 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-13 17:00:27,806 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-13 17:00:27,899 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/hbase.id with ID: 857a3f28-765b-4b71-b755-36aa53acbc62 2023-02-13 17:00:27,941 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:00:27,955 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:00:28,004 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x13600ff0 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:28,039 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78b8e196, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:28,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:00:28,067 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 17:00:28,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-13 17:00:28,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-13 17:00:28,088 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:00:28,093 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:00:28,094 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:00:28,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/data/master/store-tmp 2023-02-13 17:00:28,163 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:28,164 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 17:00:28,164 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:00:28,164 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:00:28,164 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 17:00:28,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:00:28,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:00:28,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676307628164Disabling compacts and flushes for region at 1676307628164Disabling writes for close at 1676307628165 (+1 ms)Writing region close event to WAL at 1676307628165Closed at 1676307628165 2023-02-13 17:00:28,166 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/WALs/jenkins-hbase4.apache.org,40255,1676307626519 2023-02-13 17:00:28,186 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40255%2C1676307626519, suffix=, logDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/WALs/jenkins-hbase4.apache.org,40255,1676307626519, archiveDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/oldWALs, maxLogs=10 2023-02-13 17:00:28,240 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35743,DS-a9bef184-a9e0-4522-8f8b-4566832587af,DISK] 2023-02-13 17:00:28,240 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33863,DS-4e08a395-c2a8-40ed-b610-751c45b11e60,DISK] 2023-02-13 17:00:28,240 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38629,DS-0430e927-fcbc-4f59-aa64-7d0960932709,DISK] 2023-02-13 17:00:28,249 DEBUG [RS-EventLoopGroup-5-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:00:28,317 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/WALs/jenkins-hbase4.apache.org,40255,1676307626519/jenkins-hbase4.apache.org%2C40255%2C1676307626519.1676307628196 2023-02-13 17:00:28,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35743,DS-a9bef184-a9e0-4522-8f8b-4566832587af,DISK], DatanodeInfoWithStorage[127.0.0.1:33863,DS-4e08a395-c2a8-40ed-b610-751c45b11e60,DISK], DatanodeInfoWithStorage[127.0.0.1:38629,DS-0430e927-fcbc-4f59-aa64-7d0960932709,DISK]] 2023-02-13 17:00:28,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:00:28,319 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:28,322 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:00:28,323 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:00:28,383 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:00:28,391 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 17:00:28,415 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 17:00:28,427 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:28,435 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:00:28,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:00:28,451 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:00:28,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:28,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65862282, jitterRate=-0.018575519323349}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:28,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676307628349Initializing all the Stores at 1676307628351 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307628352 (+1 ms)Cleaning up temporary data from old regions at 1676307628442 (+90 ms)Cleaning up detritus from prior splits at 1676307628443 (+1 ms)Region opened successfully at 1676307628456 (+13 ms) 2023-02-13 17:00:28,457 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 17:00:28,476 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 17:00:28,476 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 17:00:28,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 17:00:28,480 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-13 17:00:28,511 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-13 17:00:28,511 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 17:00:28,537 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 17:00:28,542 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 17:00:28,565 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 17:00:28,569 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 17:00:28,570 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 17:00:28,575 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 17:00:28,579 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 17:00:28,582 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:00:28,583 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 17:00:28,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 17:00:28,596 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 17:00:28,600 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:00:28,601 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:00:28,601 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:00:28,601 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:00:28,601 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:00:28,601 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,40255,1676307626519, sessionid=0x1004ea060850000, setting cluster-up flag (Was=false) 2023-02-13 17:00:28,620 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:00:28,626 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 17:00:28,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40255,1676307626519 2023-02-13 17:00:28,633 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:00:28,640 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 17:00:28,641 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40255,1676307626519 2023-02-13 17:00:28,643 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp 2023-02-13 17:00:28,666 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(952): ClusterId : 857a3f28-765b-4b71-b755-36aa53acbc62 2023-02-13 17:00:28,666 INFO [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(952): ClusterId : 857a3f28-765b-4b71-b755-36aa53acbc62 2023-02-13 17:00:28,666 INFO [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(952): ClusterId : 857a3f28-765b-4b71-b755-36aa53acbc62 2023-02-13 17:00:28,672 DEBUG [RS:2;jenkins-hbase4:34035] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:00:28,672 DEBUG [RS:0;jenkins-hbase4:43327] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:00:28,672 DEBUG [RS:1;jenkins-hbase4:43065] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:00:28,678 DEBUG [RS:0;jenkins-hbase4:43327] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:00:28,678 DEBUG [RS:1;jenkins-hbase4:43065] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:00:28,678 DEBUG [RS:2;jenkins-hbase4:34035] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:00:28,678 DEBUG [RS:1;jenkins-hbase4:43065] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:00:28,678 DEBUG [RS:0;jenkins-hbase4:43327] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:00:28,678 DEBUG [RS:2;jenkins-hbase4:34035] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:00:28,683 DEBUG [RS:2;jenkins-hbase4:34035] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:00:28,683 DEBUG [RS:1;jenkins-hbase4:43065] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:00:28,683 DEBUG [RS:0;jenkins-hbase4:43327] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:00:28,685 DEBUG [RS:2;jenkins-hbase4:34035] zookeeper.ReadOnlyZKClient(139): Connect 0x77c0cd29 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:28,685 DEBUG [RS:0;jenkins-hbase4:43327] zookeeper.ReadOnlyZKClient(139): Connect 0x4e5196ac to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:28,685 DEBUG [RS:1;jenkins-hbase4:43065] zookeeper.ReadOnlyZKClient(139): Connect 0x450b6030 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:28,711 DEBUG [RS:1;jenkins-hbase4:43065] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bcc96e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:28,711 DEBUG [RS:0;jenkins-hbase4:43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40966116, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:28,712 DEBUG [RS:0;jenkins-hbase4:43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44a9fddc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:00:28,711 DEBUG [RS:2;jenkins-hbase4:34035] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79072da1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:28,711 DEBUG [RS:1;jenkins-hbase4:43065] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ff30b40, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:00:28,712 DEBUG [RS:2;jenkins-hbase4:34035] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d38a659, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:00:28,735 DEBUG [RS:1;jenkins-hbase4:43065] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:43065 2023-02-13 17:00:28,738 INFO [RS:1;jenkins-hbase4:43065] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:00:28,738 INFO [RS:1;jenkins-hbase4:43065] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:00:28,738 DEBUG [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:00:28,739 DEBUG [RS:0;jenkins-hbase4:43327] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:43327 2023-02-13 17:00:28,739 DEBUG [RS:2;jenkins-hbase4:34035] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:34035 2023-02-13 17:00:28,739 INFO [RS:0;jenkins-hbase4:43327] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:00:28,741 INFO [RS:0;jenkins-hbase4:43327] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:00:28,741 DEBUG [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:00:28,741 INFO [RS:2;jenkins-hbase4:34035] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:00:28,741 INFO [RS:2;jenkins-hbase4:34035] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:00:28,741 DEBUG [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:00:28,743 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40255,1676307626519 with isa=jenkins-hbase4.apache.org/172.31.14.131:34035, startcode=1676307627747 2023-02-13 17:00:28,743 INFO [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40255,1676307626519 with isa=jenkins-hbase4.apache.org/172.31.14.131:43065, startcode=1676307627716 2023-02-13 17:00:28,743 INFO [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40255,1676307626519 with isa=jenkins-hbase4.apache.org/172.31.14.131:43327, startcode=1676307627670 2023-02-13 17:00:28,761 DEBUG [RS:1;jenkins-hbase4:43065] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:00:28,761 DEBUG [RS:0;jenkins-hbase4:43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:00:28,761 DEBUG [RS:2;jenkins-hbase4:34035] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:00:28,808 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 17:00:28,816 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55649, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:00:28,816 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52555, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:00:28,816 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40913, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:00:28,820 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:00:28,820 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:00:28,821 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:00:28,822 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:00:28,822 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 17:00:28,822 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:28,822 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:00:28,822 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:28,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40255] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:28,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40255] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:28,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:28,831 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676307658831 2023-02-13 17:00:28,834 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 17:00:28,838 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 17:00:28,839 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 17:00:28,845 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 17:00:28,847 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 17:00:28,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 17:00:28,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 17:00:28,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 17:00:28,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 17:00:28,855 DEBUG [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 17:00:28,856 DEBUG [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 17:00:28,856 DEBUG [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 17:00:28,856 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:28,857 WARN [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 17:00:28,856 WARN [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 17:00:28,856 WARN [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 17:00:28,858 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 17:00:28,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 17:00:28,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 17:00:28,864 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 17:00:28,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 17:00:28,867 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307628866,5,FailOnTimeoutGroup] 2023-02-13 17:00:28,867 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307628867,5,FailOnTimeoutGroup] 2023-02-13 17:00:28,867 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:28,867 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 17:00:28,869 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:28,869 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:28,895 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 17:00:28,896 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 17:00:28,896 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5 2023-02-13 17:00:28,919 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:28,921 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 17:00:28,924 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/info 2023-02-13 17:00:28,924 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 17:00:28,925 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:28,925 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 17:00:28,928 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:00:28,928 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 17:00:28,929 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:28,929 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 17:00:28,931 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/table 2023-02-13 17:00:28,931 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 17:00:28,932 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:28,933 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740 2023-02-13 17:00:28,934 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740 2023-02-13 17:00:28,938 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 17:00:28,939 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 17:00:28,942 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:28,943 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70513336, jitterRate=0.05073058605194092}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 17:00:28,943 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676307628919Initializing all the Stores at 1676307628920 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307628921 (+1 ms)Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307628921Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307628921Cleaning up temporary data from old regions at 1676307628936 (+15 ms)Cleaning up detritus from prior splits at 1676307628936Region opened successfully at 1676307628943 (+7 ms) 2023-02-13 17:00:28,943 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 17:00:28,943 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 17:00:28,944 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 17:00:28,944 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 17:00:28,944 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 17:00:28,944 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 17:00:28,945 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676307628943Disabling compacts and flushes for region at 1676307628943Disabling writes for close at 1676307628944 (+1 ms)Writing region close event to WAL at 1676307628944Closed at 1676307628944 2023-02-13 17:00:28,950 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 17:00:28,950 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 17:00:28,958 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40255,1676307626519 with isa=jenkins-hbase4.apache.org/172.31.14.131:34035, startcode=1676307627747 2023-02-13 17:00:28,958 INFO [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40255,1676307626519 with isa=jenkins-hbase4.apache.org/172.31.14.131:43065, startcode=1676307627716 2023-02-13 17:00:28,958 INFO [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40255,1676307626519 with isa=jenkins-hbase4.apache.org/172.31.14.131:43327, startcode=1676307627670 2023-02-13 17:00:28,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 17:00:28,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:28,965 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40255] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:28,966 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40255] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:28,971 DEBUG [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5 2023-02-13 17:00:28,972 DEBUG [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5 2023-02-13 17:00:28,972 DEBUG [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5 2023-02-13 17:00:28,972 DEBUG [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45307 2023-02-13 17:00:28,972 DEBUG [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45307 2023-02-13 17:00:28,972 DEBUG [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45307 2023-02-13 17:00:28,973 DEBUG [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:00:28,972 DEBUG [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:00:28,973 DEBUG [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:00:28,978 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 17:00:28,982 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:28,983 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:00:28,985 DEBUG [RS:0;jenkins-hbase4:43327] zookeeper.ZKUtil(162): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:28,985 DEBUG [RS:2;jenkins-hbase4:34035] zookeeper.ZKUtil(162): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:28,985 WARN [RS:0;jenkins-hbase4:43327] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:00:28,985 WARN [RS:2;jenkins-hbase4:34035] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:00:28,985 INFO [RS:0;jenkins-hbase4:43327] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:00:28,986 DEBUG [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:28,986 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34035,1676307627747] 2023-02-13 17:00:28,986 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43065,1676307627716] 2023-02-13 17:00:28,986 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43327,1676307627670] 2023-02-13 17:00:28,985 INFO [RS:2;jenkins-hbase4:34035] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:00:28,986 DEBUG [RS:1;jenkins-hbase4:43065] zookeeper.ZKUtil(162): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:28,987 DEBUG [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:28,987 WARN [RS:1;jenkins-hbase4:43065] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:00:28,987 INFO [RS:1;jenkins-hbase4:43065] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:00:28,987 DEBUG [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:29,002 DEBUG [RS:2;jenkins-hbase4:34035] zookeeper.ZKUtil(162): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:29,002 DEBUG [RS:0;jenkins-hbase4:43327] zookeeper.ZKUtil(162): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:29,002 DEBUG [RS:1;jenkins-hbase4:43065] zookeeper.ZKUtil(162): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:29,003 DEBUG [RS:2;jenkins-hbase4:34035] zookeeper.ZKUtil(162): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:29,003 DEBUG [RS:0;jenkins-hbase4:43327] zookeeper.ZKUtil(162): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:29,003 DEBUG [RS:1;jenkins-hbase4:43065] zookeeper.ZKUtil(162): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:29,003 DEBUG [RS:2;jenkins-hbase4:34035] zookeeper.ZKUtil(162): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:29,004 DEBUG [RS:0;jenkins-hbase4:43327] zookeeper.ZKUtil(162): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:29,004 DEBUG [RS:1;jenkins-hbase4:43065] zookeeper.ZKUtil(162): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:29,013 DEBUG [RS:1;jenkins-hbase4:43065] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:00:29,013 DEBUG [RS:2;jenkins-hbase4:34035] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:00:29,013 DEBUG [RS:0;jenkins-hbase4:43327] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:00:29,022 INFO [RS:0;jenkins-hbase4:43327] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:00:29,022 INFO [RS:1;jenkins-hbase4:43065] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:00:29,023 INFO [RS:2;jenkins-hbase4:34035] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:00:29,043 INFO [RS:0;jenkins-hbase4:43327] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:00:29,043 INFO [RS:1;jenkins-hbase4:43065] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:00:29,043 INFO [RS:2;jenkins-hbase4:34035] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:00:29,046 INFO [RS:2;jenkins-hbase4:34035] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:00:29,046 INFO [RS:0;jenkins-hbase4:43327] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:00:29,046 INFO [RS:1;jenkins-hbase4:43065] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:00:29,047 INFO [RS:0;jenkins-hbase4:43327] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,047 INFO [RS:2;jenkins-hbase4:34035] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,047 INFO [RS:1;jenkins-hbase4:43065] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,048 INFO [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:00:29,048 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:00:29,048 INFO [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:00:29,056 INFO [RS:1;jenkins-hbase4:43065] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,056 INFO [RS:0;jenkins-hbase4:43327] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,056 INFO [RS:2;jenkins-hbase4:34035] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,056 DEBUG [RS:1;jenkins-hbase4:43065] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:0;jenkins-hbase4:43327] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:1;jenkins-hbase4:43065] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:2;jenkins-hbase4:34035] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:1;jenkins-hbase4:43065] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:2;jenkins-hbase4:34035] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:1;jenkins-hbase4:43065] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:2;jenkins-hbase4:34035] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:1;jenkins-hbase4:43065] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:2;jenkins-hbase4:34035] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:1;jenkins-hbase4:43065] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:00:29,057 DEBUG [RS:2;jenkins-hbase4:34035] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:1;jenkins-hbase4:43065] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:2;jenkins-hbase4:34035] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:00:29,057 DEBUG [RS:1;jenkins-hbase4:43065] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:2;jenkins-hbase4:34035] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,058 DEBUG [RS:1;jenkins-hbase4:43065] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,058 DEBUG [RS:2;jenkins-hbase4:34035] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,058 DEBUG [RS:1;jenkins-hbase4:43065] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,058 DEBUG [RS:2;jenkins-hbase4:34035] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,057 DEBUG [RS:0;jenkins-hbase4:43327] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,058 DEBUG [RS:2;jenkins-hbase4:34035] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,058 DEBUG [RS:0;jenkins-hbase4:43327] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,059 DEBUG [RS:0;jenkins-hbase4:43327] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,059 DEBUG [RS:0;jenkins-hbase4:43327] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,059 DEBUG [RS:0;jenkins-hbase4:43327] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:00:29,059 DEBUG [RS:0;jenkins-hbase4:43327] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,060 DEBUG [RS:0;jenkins-hbase4:43327] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,060 DEBUG [RS:0;jenkins-hbase4:43327] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,060 DEBUG [RS:0;jenkins-hbase4:43327] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:00:29,060 INFO [RS:1;jenkins-hbase4:43065] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,060 INFO [RS:1;jenkins-hbase4:43065] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,060 INFO [RS:1;jenkins-hbase4:43065] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,064 INFO [RS:2;jenkins-hbase4:34035] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,064 INFO [RS:2;jenkins-hbase4:34035] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,064 INFO [RS:2;jenkins-hbase4:34035] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,067 INFO [RS:0;jenkins-hbase4:43327] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,067 INFO [RS:0;jenkins-hbase4:43327] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,067 INFO [RS:0;jenkins-hbase4:43327] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,077 INFO [RS:1;jenkins-hbase4:43065] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:00:29,078 INFO [RS:2;jenkins-hbase4:34035] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:00:29,081 INFO [RS:2;jenkins-hbase4:34035] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34035,1676307627747-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,081 INFO [RS:1;jenkins-hbase4:43065] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43065,1676307627716-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,081 INFO [RS:0;jenkins-hbase4:43327] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:00:29,081 INFO [RS:0;jenkins-hbase4:43327] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43327,1676307627670-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,102 INFO [RS:2;jenkins-hbase4:34035] regionserver.Replication(203): jenkins-hbase4.apache.org,34035,1676307627747 started 2023-02-13 17:00:29,102 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34035,1676307627747, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34035, sessionid=0x1004ea060850003 2023-02-13 17:00:29,102 INFO [RS:0;jenkins-hbase4:43327] regionserver.Replication(203): jenkins-hbase4.apache.org,43327,1676307627670 started 2023-02-13 17:00:29,102 DEBUG [RS:2;jenkins-hbase4:34035] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:00:29,102 INFO [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,43327,1676307627670, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43327, sessionid=0x1004ea060850001 2023-02-13 17:00:29,102 DEBUG [RS:2;jenkins-hbase4:34035] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:29,102 DEBUG [RS:0;jenkins-hbase4:43327] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:00:29,102 DEBUG [RS:2;jenkins-hbase4:34035] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34035,1676307627747' 2023-02-13 17:00:29,102 DEBUG [RS:0;jenkins-hbase4:43327] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:29,103 DEBUG [RS:2;jenkins-hbase4:34035] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:00:29,103 DEBUG [RS:0;jenkins-hbase4:43327] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43327,1676307627670' 2023-02-13 17:00:29,104 DEBUG [RS:0;jenkins-hbase4:43327] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:00:29,105 DEBUG [RS:2;jenkins-hbase4:34035] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:00:29,105 DEBUG [RS:0;jenkins-hbase4:43327] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:00:29,105 INFO [RS:1;jenkins-hbase4:43065] regionserver.Replication(203): jenkins-hbase4.apache.org,43065,1676307627716 started 2023-02-13 17:00:29,105 INFO [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,43065,1676307627716, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43065, sessionid=0x1004ea060850002 2023-02-13 17:00:29,105 DEBUG [RS:1;jenkins-hbase4:43065] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:00:29,105 DEBUG [RS:1;jenkins-hbase4:43065] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:29,105 DEBUG [RS:1;jenkins-hbase4:43065] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43065,1676307627716' 2023-02-13 17:00:29,105 DEBUG [RS:0;jenkins-hbase4:43327] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:00:29,105 DEBUG [RS:1;jenkins-hbase4:43065] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:00:29,105 DEBUG [RS:2;jenkins-hbase4:34035] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:00:29,105 DEBUG [RS:0;jenkins-hbase4:43327] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:00:29,106 DEBUG [RS:2;jenkins-hbase4:34035] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:00:29,106 DEBUG [RS:2;jenkins-hbase4:34035] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:29,106 DEBUG [RS:0;jenkins-hbase4:43327] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:29,106 DEBUG [RS:2;jenkins-hbase4:34035] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34035,1676307627747' 2023-02-13 17:00:29,106 DEBUG [RS:1;jenkins-hbase4:43065] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:00:29,106 DEBUG [RS:0;jenkins-hbase4:43327] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43327,1676307627670' 2023-02-13 17:00:29,106 DEBUG [RS:2;jenkins-hbase4:34035] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:00:29,106 DEBUG [RS:0;jenkins-hbase4:43327] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:00:29,107 DEBUG [RS:1;jenkins-hbase4:43065] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:00:29,107 DEBUG [RS:1;jenkins-hbase4:43065] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:00:29,107 DEBUG [RS:0;jenkins-hbase4:43327] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:00:29,107 DEBUG [RS:1;jenkins-hbase4:43065] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:29,107 DEBUG [RS:2;jenkins-hbase4:34035] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:00:29,107 DEBUG [RS:1;jenkins-hbase4:43065] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43065,1676307627716' 2023-02-13 17:00:29,107 DEBUG [RS:1;jenkins-hbase4:43065] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:00:29,107 DEBUG [RS:0;jenkins-hbase4:43327] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:00:29,107 DEBUG [RS:2;jenkins-hbase4:34035] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:00:29,108 DEBUG [RS:1;jenkins-hbase4:43065] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:00:29,108 INFO [RS:0;jenkins-hbase4:43327] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:00:29,108 INFO [RS:2;jenkins-hbase4:34035] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:00:29,108 INFO [RS:2;jenkins-hbase4:34035] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:00:29,108 INFO [RS:0;jenkins-hbase4:43327] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:00:29,108 DEBUG [RS:1;jenkins-hbase4:43065] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:00:29,108 INFO [RS:1;jenkins-hbase4:43065] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:00:29,108 INFO [RS:1;jenkins-hbase4:43065] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:00:29,134 DEBUG [jenkins-hbase4:40255] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 17:00:29,138 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:00:29,144 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:00:29,144 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:00:29,144 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:00:29,144 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:00:29,147 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34035,1676307627747, state=OPENING 2023-02-13 17:00:29,154 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 17:00:29,156 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:00:29,156 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 17:00:29,161 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:29,219 INFO [RS:1;jenkins-hbase4:43065] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43065%2C1676307627716, suffix=, logDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,43065,1676307627716, archiveDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/oldWALs, maxLogs=32 2023-02-13 17:00:29,219 INFO [RS:2;jenkins-hbase4:34035] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34035%2C1676307627747, suffix=, logDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,34035,1676307627747, archiveDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/oldWALs, maxLogs=32 2023-02-13 17:00:29,219 INFO [RS:0;jenkins-hbase4:43327] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43327%2C1676307627670, suffix=, logDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,43327,1676307627670, archiveDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/oldWALs, maxLogs=32 2023-02-13 17:00:29,254 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38629,DS-0430e927-fcbc-4f59-aa64-7d0960932709,DISK] 2023-02-13 17:00:29,254 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35743,DS-a9bef184-a9e0-4522-8f8b-4566832587af,DISK] 2023-02-13 17:00:29,261 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33863,DS-4e08a395-c2a8-40ed-b610-751c45b11e60,DISK] 2023-02-13 17:00:29,261 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38629,DS-0430e927-fcbc-4f59-aa64-7d0960932709,DISK] 2023-02-13 17:00:29,268 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33863,DS-4e08a395-c2a8-40ed-b610-751c45b11e60,DISK] 2023-02-13 17:00:29,268 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35743,DS-a9bef184-a9e0-4522-8f8b-4566832587af,DISK] 2023-02-13 17:00:29,268 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33863,DS-4e08a395-c2a8-40ed-b610-751c45b11e60,DISK] 2023-02-13 17:00:29,269 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38629,DS-0430e927-fcbc-4f59-aa64-7d0960932709,DISK] 2023-02-13 17:00:29,269 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35743,DS-a9bef184-a9e0-4522-8f8b-4566832587af,DISK] 2023-02-13 17:00:29,280 INFO [RS:0;jenkins-hbase4:43327] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,43327,1676307627670/jenkins-hbase4.apache.org%2C43327%2C1676307627670.1676307629223 2023-02-13 17:00:29,280 INFO [RS:1;jenkins-hbase4:43065] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,43065,1676307627716/jenkins-hbase4.apache.org%2C43065%2C1676307627716.1676307629223 2023-02-13 17:00:29,280 DEBUG [RS:0;jenkins-hbase4:43327] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38629,DS-0430e927-fcbc-4f59-aa64-7d0960932709,DISK], DatanodeInfoWithStorage[127.0.0.1:35743,DS-a9bef184-a9e0-4522-8f8b-4566832587af,DISK], DatanodeInfoWithStorage[127.0.0.1:33863,DS-4e08a395-c2a8-40ed-b610-751c45b11e60,DISK]] 2023-02-13 17:00:29,280 INFO [RS:2;jenkins-hbase4:34035] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,34035,1676307627747/jenkins-hbase4.apache.org%2C34035%2C1676307627747.1676307629223 2023-02-13 17:00:29,281 DEBUG [RS:1;jenkins-hbase4:43065] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35743,DS-a9bef184-a9e0-4522-8f8b-4566832587af,DISK], DatanodeInfoWithStorage[127.0.0.1:33863,DS-4e08a395-c2a8-40ed-b610-751c45b11e60,DISK], DatanodeInfoWithStorage[127.0.0.1:38629,DS-0430e927-fcbc-4f59-aa64-7d0960932709,DISK]] 2023-02-13 17:00:29,281 DEBUG [RS:2;jenkins-hbase4:34035] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35743,DS-a9bef184-a9e0-4522-8f8b-4566832587af,DISK], DatanodeInfoWithStorage[127.0.0.1:38629,DS-0430e927-fcbc-4f59-aa64-7d0960932709,DISK], DatanodeInfoWithStorage[127.0.0.1:33863,DS-4e08a395-c2a8-40ed-b610-751c45b11e60,DISK]] 2023-02-13 17:00:29,345 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:29,348 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 17:00:29,350 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59368, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 17:00:29,364 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 17:00:29,364 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:00:29,368 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34035%2C1676307627747.meta, suffix=.meta, logDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,34035,1676307627747, archiveDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/oldWALs, maxLogs=32 2023-02-13 17:00:29,389 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33863,DS-4e08a395-c2a8-40ed-b610-751c45b11e60,DISK] 2023-02-13 17:00:29,390 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35743,DS-a9bef184-a9e0-4522-8f8b-4566832587af,DISK] 2023-02-13 17:00:29,391 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38629,DS-0430e927-fcbc-4f59-aa64-7d0960932709,DISK] 2023-02-13 17:00:29,395 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/WALs/jenkins-hbase4.apache.org,34035,1676307627747/jenkins-hbase4.apache.org%2C34035%2C1676307627747.meta.1676307629369.meta 2023-02-13 17:00:29,399 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33863,DS-4e08a395-c2a8-40ed-b610-751c45b11e60,DISK], DatanodeInfoWithStorage[127.0.0.1:35743,DS-a9bef184-a9e0-4522-8f8b-4566832587af,DISK], DatanodeInfoWithStorage[127.0.0.1:38629,DS-0430e927-fcbc-4f59-aa64-7d0960932709,DISK]] 2023-02-13 17:00:29,400 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:00:29,401 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 17:00:29,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 17:00:29,426 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 17:00:29,431 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 17:00:29,431 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:29,431 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 17:00:29,431 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 17:00:29,434 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 17:00:29,436 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/info 2023-02-13 17:00:29,436 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/info 2023-02-13 17:00:29,437 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 17:00:29,438 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:29,438 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 17:00:29,439 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:00:29,439 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:00:29,440 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 17:00:29,441 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:29,441 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 17:00:29,442 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/table 2023-02-13 17:00:29,442 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/table 2023-02-13 17:00:29,443 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 17:00:29,443 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:29,445 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740 2023-02-13 17:00:29,448 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740 2023-02-13 17:00:29,451 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 17:00:29,454 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 17:00:29,455 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67080721, jitterRate=-4.193633794784546E-4}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 17:00:29,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676307629432Writing region info on filesystem at 1676307629432Initializing all the Stores at 1676307629434 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307629434Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307629434Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307629434Cleaning up temporary data from old regions at 1676307629450 (+16 ms)Cleaning up detritus from prior splits at 1676307629450Running coprocessor post-open hooks at 1676307629456 (+6 ms)Region opened successfully at 1676307629456 2023-02-13 17:00:29,466 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676307629337 2023-02-13 17:00:29,484 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 17:00:29,485 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 17:00:29,486 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34035,1676307627747, state=OPEN 2023-02-13 17:00:29,489 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 17:00:29,489 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 17:00:29,494 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 17:00:29,494 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34035,1676307627747 in 328 msec 2023-02-13 17:00:29,499 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 17:00:29,499 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 535 msec 2023-02-13 17:00:29,504 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 784 msec 2023-02-13 17:00:29,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676307627789, completionTime=-1 2023-02-13 17:00:29,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 17:00:29,505 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 17:00:29,573 DEBUG [hconnection-0x60dfe106-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:29,576 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59372, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:29,592 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 17:00:29,592 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676307689592 2023-02-13 17:00:29,592 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676307749592 2023-02-13 17:00:29,592 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 87 msec 2023-02-13 17:00:29,619 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40255,1676307626519-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,619 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40255,1676307626519-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,620 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40255,1676307626519-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,621 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:40255, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,622 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 17:00:29,628 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 17:00:29,636 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 17:00:29,638 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 17:00:29,647 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 17:00:29,649 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:00:29,652 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:00:29,673 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199 2023-02-13 17:00:29,675 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199 empty. 2023-02-13 17:00:29,676 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199 2023-02-13 17:00:29,676 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 17:00:29,709 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 17:00:29,711 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 0c94fa9feea4731bb9272766d7d4c199, NAME => 'hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:29,737 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:29,737 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 0c94fa9feea4731bb9272766d7d4c199, disabling compactions & flushes 2023-02-13 17:00:29,737 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:00:29,737 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:00:29,737 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. after waiting 0 ms 2023-02-13 17:00:29,737 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:00:29,737 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:00:29,737 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 0c94fa9feea4731bb9272766d7d4c199: Waiting for close lock at 1676307629737Disabling compacts and flushes for region at 1676307629737Disabling writes for close at 1676307629737Writing region close event to WAL at 1676307629737Closed at 1676307629737 2023-02-13 17:00:29,742 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:00:29,757 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307629745"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307629745"}]},"ts":"1676307629745"} 2023-02-13 17:00:29,784 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 17:00:29,786 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:00:29,791 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307629787"}]},"ts":"1676307629787"} 2023-02-13 17:00:29,799 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 17:00:29,804 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:00:29,805 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:00:29,805 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:00:29,805 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:00:29,805 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:00:29,808 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=0c94fa9feea4731bb9272766d7d4c199, ASSIGN}] 2023-02-13 17:00:29,811 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=0c94fa9feea4731bb9272766d7d4c199, ASSIGN 2023-02-13 17:00:29,814 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=0c94fa9feea4731bb9272766d7d4c199, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=false 2023-02-13 17:00:29,966 INFO [jenkins-hbase4:40255] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:00:29,967 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=0c94fa9feea4731bb9272766d7d4c199, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:29,968 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307629967"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307629967"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307629967"}]},"ts":"1676307629967"} 2023-02-13 17:00:29,972 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 0c94fa9feea4731bb9272766d7d4c199, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:30,137 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:00:30,137 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0c94fa9feea4731bb9272766d7d4c199, NAME => 'hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:00:30,138 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 0c94fa9feea4731bb9272766d7d4c199 2023-02-13 17:00:30,138 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:30,138 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0c94fa9feea4731bb9272766d7d4c199 2023-02-13 17:00:30,138 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0c94fa9feea4731bb9272766d7d4c199 2023-02-13 17:00:30,140 INFO [StoreOpener-0c94fa9feea4731bb9272766d7d4c199-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 0c94fa9feea4731bb9272766d7d4c199 2023-02-13 17:00:30,143 DEBUG [StoreOpener-0c94fa9feea4731bb9272766d7d4c199-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199/info 2023-02-13 17:00:30,143 DEBUG [StoreOpener-0c94fa9feea4731bb9272766d7d4c199-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199/info 2023-02-13 17:00:30,143 INFO [StoreOpener-0c94fa9feea4731bb9272766d7d4c199-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0c94fa9feea4731bb9272766d7d4c199 columnFamilyName info 2023-02-13 17:00:30,144 INFO [StoreOpener-0c94fa9feea4731bb9272766d7d4c199-1] regionserver.HStore(310): Store=0c94fa9feea4731bb9272766d7d4c199/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:30,146 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199 2023-02-13 17:00:30,147 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199 2023-02-13 17:00:30,150 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0c94fa9feea4731bb9272766d7d4c199 2023-02-13 17:00:30,153 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:30,154 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0c94fa9feea4731bb9272766d7d4c199; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66017570, jitterRate=-0.016261547803878784}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:30,154 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0c94fa9feea4731bb9272766d7d4c199: Running coprocessor pre-open hook at 1676307630139Writing region info on filesystem at 1676307630139Initializing all the Stores at 1676307630140 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307630140Cleaning up temporary data from old regions at 1676307630147 (+7 ms)Cleaning up detritus from prior splits at 1676307630148 (+1 ms)Running coprocessor post-open hooks at 1676307630154 (+6 ms)Region opened successfully at 1676307630154 2023-02-13 17:00:30,156 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199., pid=6, masterSystemTime=1676307630126 2023-02-13 17:00:30,160 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:00:30,160 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:00:30,162 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=0c94fa9feea4731bb9272766d7d4c199, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:30,163 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307630161"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307630161"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307630161"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307630161"}]},"ts":"1676307630161"} 2023-02-13 17:00:30,169 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 17:00:30,169 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 0c94fa9feea4731bb9272766d7d4c199, server=jenkins-hbase4.apache.org,34035,1676307627747 in 194 msec 2023-02-13 17:00:30,173 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 17:00:30,173 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=0c94fa9feea4731bb9272766d7d4c199, ASSIGN in 361 msec 2023-02-13 17:00:30,175 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:00:30,175 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307630175"}]},"ts":"1676307630175"} 2023-02-13 17:00:30,177 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 17:00:30,180 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:00:30,183 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 541 msec 2023-02-13 17:00:30,250 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 17:00:30,251 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 17:00:30,251 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:00:30,285 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 17:00:30,304 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 17:00:30,312 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 33 msec 2023-02-13 17:00:30,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 17:00:30,330 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 17:00:30,336 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-02-13 17:00:30,348 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 17:00:30,350 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 17:00:30,350 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.549sec 2023-02-13 17:00:30,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 17:00:30,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 17:00:30,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 17:00:30,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40255,1676307626519-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 17:00:30,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40255,1676307626519-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 17:00:30,366 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 17:00:30,373 DEBUG [Listener at localhost/40167] zookeeper.ReadOnlyZKClient(139): Connect 0x1c633018 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:30,380 DEBUG [Listener at localhost/40167] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b210801, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:30,394 DEBUG [hconnection-0x7b9b1635-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:30,407 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59374, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:30,416 INFO [Listener at localhost/40167] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,40255,1676307626519 2023-02-13 17:00:30,444 INFO [Listener at localhost/40167] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=357, OpenFileDescriptor=693, MaxFileDescriptor=60000, SystemLoadAverage=244, ProcessCount=168, AvailableMemoryMB=1405 2023-02-13 17:00:30,458 DEBUG [Listener at localhost/40167] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 17:00:30,461 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33796, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 17:00:30,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-13 17:00:30,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:30,479 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676307630447 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:00:30,481 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676307630447 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:00:30,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676307630447" procId is: 9 2023-02-13 17:00:30,490 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:30,490 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:30,490 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:30,490 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:30,491 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:30,491 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 empty. 2023-02-13 17:00:30,491 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:30,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:00:30,492 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b empty. 2023-02-13 17:00:30,492 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 empty. 2023-02-13 17:00:30,494 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 empty. 2023-02-13 17:00:30,494 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 empty. 2023-02-13 17:00:30,494 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:30,495 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea empty. 2023-02-13 17:00:30,495 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:30,495 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:30,495 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:30,496 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:30,496 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:30,496 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676307630447 regions 2023-02-13 17:00:30,519 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000001 2023-02-13 17:00:30,521 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(7671): creating {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:30,549 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(7671): creating {ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:30,549 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(7671): creating {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:30,591 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:30,592 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1603): Closing 0e16e4423db3157675739f103f8f5a30, disabling compactions & flushes 2023-02-13 17:00:30,592 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:30,592 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:30,592 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. after waiting 0 ms 2023-02-13 17:00:30,592 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:30,592 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:30,592 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:30,592 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1557): Region close journal for 0e16e4423db3157675739f103f8f5a30: Waiting for close lock at 1676307630592Disabling compacts and flushes for region at 1676307630592Disabling writes for close at 1676307630592Writing region close event to WAL at 1676307630592Closed at 1676307630592 2023-02-13 17:00:30,594 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1603): Closing bb0e3d66ca9f4f9ce3e87dc8bfc74c30, disabling compactions & flushes 2023-02-13 17:00:30,595 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:30,595 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:30,595 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. after waiting 0 ms 2023-02-13 17:00:30,595 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(7671): creating {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:30,595 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:30,595 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:30,595 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1557): Region close journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Waiting for close lock at 1676307630594Disabling compacts and flushes for region at 1676307630594Disabling writes for close at 1676307630595 (+1 ms)Writing region close event to WAL at 1676307630595Closed at 1676307630595 2023-02-13 17:00:30,596 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(7671): creating {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:30,596 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:30,596 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1603): Closing 2457c5d248f99f39cff76beaa71b4345, disabling compactions & flushes 2023-02-13 17:00:30,596 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:30,596 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:30,596 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. after waiting 0 ms 2023-02-13 17:00:30,596 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:30,597 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:30,597 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1557): Region close journal for 2457c5d248f99f39cff76beaa71b4345: Waiting for close lock at 1676307630596Disabling compacts and flushes for region at 1676307630596Disabling writes for close at 1676307630596Writing region close event to WAL at 1676307630597 (+1 ms)Closed at 1676307630597 2023-02-13 17:00:30,597 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(7671): creating {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:30,621 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:30,622 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1603): Closing f31ed543f552e93833ffd6a17e17334b, disabling compactions & flushes 2023-02-13 17:00:30,622 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:30,622 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:30,622 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. after waiting 0 ms 2023-02-13 17:00:30,623 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:30,623 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:30,623 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:30,623 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-0] regionserver.HRegion(1557): Region close journal for f31ed543f552e93833ffd6a17e17334b: Waiting for close lock at 1676307630622Disabling compacts and flushes for region at 1676307630622Disabling writes for close at 1676307630623 (+1 ms)Writing region close event to WAL at 1676307630623Closed at 1676307630623 2023-02-13 17:00:30,625 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1603): Closing 7925c8f572739336270bf824c12623ea, disabling compactions & flushes 2023-02-13 17:00:30,625 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:30,625 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:30,625 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1603): Closing 61523f42895e23c1110bbb6a6531b0a6, disabling compactions & flushes 2023-02-13 17:00:30,625 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:30,625 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:30,625 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. after waiting 0 ms 2023-02-13 17:00:30,625 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:30,626 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:30,626 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. after waiting 0 ms 2023-02-13 17:00:30,626 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:30,626 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:30,626 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-2] regionserver.HRegion(1557): Region close journal for 7925c8f572739336270bf824c12623ea: Waiting for close lock at 1676307630625Disabling compacts and flushes for region at 1676307630625Disabling writes for close at 1676307630625Writing region close event to WAL at 1676307630626 (+1 ms)Closed at 1676307630626 2023-02-13 17:00:30,626 INFO [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:30,626 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676307630447-pool-1] regionserver.HRegion(1557): Region close journal for 61523f42895e23c1110bbb6a6531b0a6: Waiting for close lock at 1676307630625Disabling compacts and flushes for region at 1676307630625Disabling writes for close at 1676307630626 (+1 ms)Writing region close event to WAL at 1676307630626Closed at 1676307630626 2023-02-13 17:00:30,631 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676307630447 execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:00:30,632 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307630632"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307630632"}]},"ts":"1676307630632"} 2023-02-13 17:00:30,633 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307630632"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307630632"}]},"ts":"1676307630632"} 2023-02-13 17:00:30,633 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307630632"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307630632"}]},"ts":"1676307630632"} 2023-02-13 17:00:30,633 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307630632"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307630632"}]},"ts":"1676307630632"} 2023-02-13 17:00:30,633 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307630632"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307630632"}]},"ts":"1676307630632"} 2023-02-13 17:00:30,633 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307630632"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307630632"}]},"ts":"1676307630632"} 2023-02-13 17:00:30,678 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 17:00:30,680 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676307630447 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:00:30,681 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307630680"}]},"ts":"1676307630680"} 2023-02-13 17:00:30,683 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLING in hbase:meta 2023-02-13 17:00:30,688 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:00:30,692 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:00:30,692 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:00:30,692 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:00:30,692 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:00:30,693 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN}] 2023-02-13 17:00:30,696 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN 2023-02-13 17:00:30,696 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN 2023-02-13 17:00:30,697 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN 2023-02-13 17:00:30,697 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN 2023-02-13 17:00:30,698 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN 2023-02-13 17:00:30,699 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=false 2023-02-13 17:00:30,699 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=false 2023-02-13 17:00:30,699 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=false 2023-02-13 17:00:30,698 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=false 2023-02-13 17:00:30,700 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=false 2023-02-13 17:00:30,700 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN 2023-02-13 17:00:30,701 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=false 2023-02-13 17:00:30,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:00:30,849 INFO [jenkins-hbase4:40255] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 17:00:30,851 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:30,851 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:30,851 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:30,851 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:30,851 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:30,852 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307630851"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307630851"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307630851"}]},"ts":"1676307630851"} 2023-02-13 17:00:30,852 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307630851"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307630851"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307630851"}]},"ts":"1676307630851"} 2023-02-13 17:00:30,852 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307630851"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307630851"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307630851"}]},"ts":"1676307630851"} 2023-02-13 17:00:30,852 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307630851"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307630851"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307630851"}]},"ts":"1676307630851"} 2023-02-13 17:00:30,852 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307630851"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307630851"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307630851"}]},"ts":"1676307630851"} 2023-02-13 17:00:30,855 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=12, state=RUNNABLE; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:30,857 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=15, state=RUNNABLE; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:30,858 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=11, state=RUNNABLE; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:30,861 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=10, state=RUNNABLE; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:30,862 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:30,863 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307630862"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307630862"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307630862"}]},"ts":"1676307630862"} 2023-02-13 17:00:30,866 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=14, state=RUNNABLE; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:30,867 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=13, state=RUNNABLE; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:31,009 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:31,009 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 17:00:31,011 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:31,012 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 17:00:31,012 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55840, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 17:00:31,015 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41978, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 17:00:31,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:31,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 17:00:31,021 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:31,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:31,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:31,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 17:00:31,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:31,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:31,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:31,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:31,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:31,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:31,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 17:00:31,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:31,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:31,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:31,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:31,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:31,025 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:31,025 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:31,025 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:31,028 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:31,028 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:31,028 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:31,028 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:31,028 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:31,028 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:31,031 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e16e4423db3157675739f103f8f5a30 columnFamilyName cf 2023-02-13 17:00:31,031 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 columnFamilyName cf 2023-02-13 17:00:31,031 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2457c5d248f99f39cff76beaa71b4345 columnFamilyName cf 2023-02-13 17:00:31,034 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(310): Store=0e16e4423db3157675739f103f8f5a30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:31,034 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(310): Store=2457c5d248f99f39cff76beaa71b4345/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:31,034 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(310): Store=bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:31,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:31,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:31,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:31,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:31,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:31,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:31,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:31,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:31,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:31,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:31,071 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bb0e3d66ca9f4f9ce3e87dc8bfc74c30; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66053815, jitterRate=-0.015721455216407776}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:31,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Running coprocessor pre-open hook at 1676307631022Writing region info on filesystem at 1676307631022Initializing all the Stores at 1676307631024 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307631024Cleaning up temporary data from old regions at 1676307631040 (+16 ms)Cleaning up detritus from prior splits at 1676307631041 (+1 ms)Running coprocessor post-open hooks at 1676307631071 (+30 ms)Region opened successfully at 1676307631071 2023-02-13 17:00:31,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:31,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30., pid=16, masterSystemTime=1676307631009 2023-02-13 17:00:31,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:31,085 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:31,085 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e16e4423db3157675739f103f8f5a30; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74408411, jitterRate=0.10877172648906708}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:31,085 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:31,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e16e4423db3157675739f103f8f5a30: Running coprocessor pre-open hook at 1676307631023Writing region info on filesystem at 1676307631023Initializing all the Stores at 1676307631025 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307631025Cleaning up temporary data from old regions at 1676307631040 (+15 ms)Cleaning up detritus from prior splits at 1676307631040Running coprocessor post-open hooks at 1676307631085 (+45 ms)Region opened successfully at 1676307631085 2023-02-13 17:00:31,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:31,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 17:00:31,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:31,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2457c5d248f99f39cff76beaa71b4345; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74251730, jitterRate=0.10643699765205383}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:31,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:31,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30., pid=19, masterSystemTime=1676307631016 2023-02-13 17:00:31,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:31,088 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:31,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:31,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2457c5d248f99f39cff76beaa71b4345: Running coprocessor pre-open hook at 1676307631023Writing region info on filesystem at 1676307631023Initializing all the Stores at 1676307631025 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307631025Cleaning up temporary data from old regions at 1676307631040 (+15 ms)Cleaning up detritus from prior splits at 1676307631041 (+1 ms)Running coprocessor post-open hooks at 1676307631087 (+46 ms)Region opened successfully at 1676307631087 2023-02-13 17:00:31,088 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631088"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307631088"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307631088"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307631088"}]},"ts":"1676307631088"} 2023-02-13 17:00:31,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345., pid=18, masterSystemTime=1676307631011 2023-02-13 17:00:31,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:31,093 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:31,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:31,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:31,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 17:00:31,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:31,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:31,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:31,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:31,094 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:31,094 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307631094"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307631094"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307631094"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307631094"}]},"ts":"1676307631094"} 2023-02-13 17:00:31,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:31,096 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:31,097 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:31,096 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:31,097 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:31,097 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:31,097 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 17:00:31,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:31,098 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:31,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:31,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:31,098 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631098"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307631098"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307631098"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307631098"}]},"ts":"1676307631098"} 2023-02-13 17:00:31,098 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7925c8f572739336270bf824c12623ea columnFamilyName cf 2023-02-13 17:00:31,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:31,100 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=12 2023-02-13 17:00:31,102 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 238 msec 2023-02-13 17:00:31,103 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(310): Store=7925c8f572739336270bf824c12623ea/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:31,103 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:31,103 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:31,103 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f31ed543f552e93833ffd6a17e17334b columnFamilyName cf 2023-02-13 17:00:31,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:31,105 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:31,106 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN in 409 msec 2023-02-13 17:00:31,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:31,107 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(310): Store=f31ed543f552e93833ffd6a17e17334b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:31,108 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=10 2023-02-13 17:00:31,108 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=10, state=SUCCESS; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 238 msec 2023-02-13 17:00:31,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:31,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:31,113 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN in 415 msec 2023-02-13 17:00:31,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:31,114 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=11 2023-02-13 17:00:31,114 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=11, state=SUCCESS; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670 in 249 msec 2023-02-13 17:00:31,115 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:31,116 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:31,116 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61523f42895e23c1110bbb6a6531b0a6 columnFamilyName cf 2023-02-13 17:00:31,117 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN in 421 msec 2023-02-13 17:00:31,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:31,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:31,118 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(310): Store=61523f42895e23c1110bbb6a6531b0a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:31,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7925c8f572739336270bf824c12623ea; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65882209, jitterRate=-0.018278583884239197}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:31,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7925c8f572739336270bf824c12623ea: Running coprocessor pre-open hook at 1676307631088Writing region info on filesystem at 1676307631088Initializing all the Stores at 1676307631091 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307631091Cleaning up temporary data from old regions at 1676307631108 (+17 ms)Cleaning up detritus from prior splits at 1676307631109 (+1 ms)Running coprocessor post-open hooks at 1676307631119 (+10 ms)Region opened successfully at 1676307631119 2023-02-13 17:00:31,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:31,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea., pid=20, masterSystemTime=1676307631009 2023-02-13 17:00:31,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:31,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:31,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f31ed543f552e93833ffd6a17e17334b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63679125, jitterRate=-0.05110709369182587}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:31,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f31ed543f552e93833ffd6a17e17334b: Running coprocessor pre-open hook at 1676307631094Writing region info on filesystem at 1676307631094Initializing all the Stores at 1676307631096 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307631096Cleaning up temporary data from old regions at 1676307631111 (+15 ms)Cleaning up detritus from prior splits at 1676307631112 (+1 ms)Running coprocessor post-open hooks at 1676307631123 (+11 ms)Region opened successfully at 1676307631123 2023-02-13 17:00:31,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:31,126 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:31,126 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b., pid=21, masterSystemTime=1676307631016 2023-02-13 17:00:31,128 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:31,128 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631127"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307631127"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307631127"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307631127"}]},"ts":"1676307631127"} 2023-02-13 17:00:31,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:31,129 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:31,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:31,131 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:31,131 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631130"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307631130"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307631130"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307631130"}]},"ts":"1676307631130"} 2023-02-13 17:00:31,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:31,135 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 61523f42895e23c1110bbb6a6531b0a6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63233632, jitterRate=-0.05774545669555664}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:31,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 61523f42895e23c1110bbb6a6531b0a6: Running coprocessor pre-open hook at 1676307631102Writing region info on filesystem at 1676307631102Initializing all the Stores at 1676307631104 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307631104Cleaning up temporary data from old regions at 1676307631123 (+19 ms)Cleaning up detritus from prior splits at 1676307631125 (+2 ms)Running coprocessor post-open hooks at 1676307631135 (+10 ms)Region opened successfully at 1676307631135 2023-02-13 17:00:31,135 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=14 2023-02-13 17:00:31,136 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=14, state=SUCCESS; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716 in 265 msec 2023-02-13 17:00:31,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6., pid=17, masterSystemTime=1676307631011 2023-02-13 17:00:31,139 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=13 2023-02-13 17:00:31,139 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=13, state=SUCCESS; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747 in 267 msec 2023-02-13 17:00:31,139 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN in 443 msec 2023-02-13 17:00:31,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:31,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:31,141 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:31,142 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307631141"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307631141"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307631141"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307631141"}]},"ts":"1676307631141"} 2023-02-13 17:00:31,142 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN in 446 msec 2023-02-13 17:00:31,146 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=15 2023-02-13 17:00:31,148 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=15, state=SUCCESS; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670 in 287 msec 2023-02-13 17:00:31,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=9 2023-02-13 17:00:31,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN in 455 msec 2023-02-13 17:00:31,153 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676307630447 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:00:31,153 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307631153"}]},"ts":"1676307631153"} 2023-02-13 17:00:31,155 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLED in hbase:meta 2023-02-13 17:00:31,160 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676307630447 execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:00:31,163 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676307630447 in 686 msec 2023-02-13 17:00:31,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:00:31,252 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 9 completed 2023-02-13 17:00:31,260 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676307630447 2023-02-13 17:00:31,261 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:31,264 INFO [Listener at localhost/40167] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:00:31,288 DEBUG [hconnection-0x7b9b1635-shared-pool-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:31,292 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41982, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:31,294 DEBUG [hconnection-0x7b9b1635-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:31,297 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55848, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:31,307 INFO [Listener at localhost/40167] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:31,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676307630447 2023-02-13 17:00:31,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:31,322 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307631322"}]},"ts":"1676307631322"} 2023-02-13 17:00:31,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 17:00:31,325 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLING in hbase:meta 2023-02-13 17:00:31,329 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676307630447 to state=DISABLING 2023-02-13 17:00:31,331 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN}] 2023-02-13 17:00:31,333 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN 2023-02-13 17:00:31,333 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN 2023-02-13 17:00:31,334 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN 2023-02-13 17:00:31,334 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN 2023-02-13 17:00:31,334 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN 2023-02-13 17:00:31,336 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:31,336 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:31,336 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:31,336 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307631335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307631335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307631335"}]},"ts":"1676307631335"} 2023-02-13 17:00:31,336 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631336"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307631336"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307631336"}]},"ts":"1676307631336"} 2023-02-13 17:00:31,336 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:31,336 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:31,336 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631336"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307631336"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307631336"}]},"ts":"1676307631336"} 2023-02-13 17:00:31,336 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631336"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307631336"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307631336"}]},"ts":"1676307631336"} 2023-02-13 17:00:31,336 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307631336"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307631336"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307631336"}]},"ts":"1676307631336"} 2023-02-13 17:00:31,338 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=23, state=RUNNABLE; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:31,341 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=24, state=RUNNABLE; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:31,342 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=25, state=RUNNABLE; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:31,344 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=26, state=RUNNABLE; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:31,346 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:31,352 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN 2023-02-13 17:00:31,353 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:31,353 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631353"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307631353"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307631353"}]},"ts":"1676307631353"} 2023-02-13 17:00:31,356 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=27, state=RUNNABLE; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:31,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:31,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:31,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 61523f42895e23c1110bbb6a6531b0a6, disabling compactions & flushes 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f31ed543f552e93833ffd6a17e17334b, disabling compactions & flushes 2023-02-13 17:00:31,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:31,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. after waiting 0 ms 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. after waiting 0 ms 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:31,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bb0e3d66ca9f4f9ce3e87dc8bfc74c30, disabling compactions & flushes 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:31,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. after waiting 0 ms 2023-02-13 17:00:31,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:31,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:31,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:31,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:31,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:31,516 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 61523f42895e23c1110bbb6a6531b0a6: Waiting for close lock at 1676307631505Running coprocessor pre-close hooks at 1676307631505Disabling compacts and flushes for region at 1676307631505Disabling writes for close at 1676307631506 (+1 ms)Writing region close event to WAL at 1676307631506Running coprocessor post-close hooks at 1676307631516 (+10 ms)Closed at 1676307631516 2023-02-13 17:00:31,516 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Waiting for close lock at 1676307631505Running coprocessor pre-close hooks at 1676307631505Disabling compacts and flushes for region at 1676307631505Disabling writes for close at 1676307631506 (+1 ms)Writing region close event to WAL at 1676307631508 (+2 ms)Running coprocessor post-close hooks at 1676307631516 (+8 ms)Closed at 1676307631516 2023-02-13 17:00:31,516 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:31,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:31,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f31ed543f552e93833ffd6a17e17334b: Waiting for close lock at 1676307631506Running coprocessor pre-close hooks at 1676307631506Disabling compacts and flushes for region at 1676307631506Disabling writes for close at 1676307631506Writing region close event to WAL at 1676307631507 (+1 ms)Running coprocessor post-close hooks at 1676307631517 (+10 ms)Closed at 1676307631518 (+1 ms) 2023-02-13 17:00:31,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:31,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:31,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7925c8f572739336270bf824c12623ea, disabling compactions & flushes 2023-02-13 17:00:31,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:31,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:31,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. after waiting 0 ms 2023-02-13 17:00:31,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:31,520 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSED 2023-02-13 17:00:31,520 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631520"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307631520"}]},"ts":"1676307631520"} 2023-02-13 17:00:31,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:31,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:31,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2457c5d248f99f39cff76beaa71b4345, disabling compactions & flushes 2023-02-13 17:00:31,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:31,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:31,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. after waiting 0 ms 2023-02-13 17:00:31,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:31,523 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSED 2023-02-13 17:00:31,524 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307631523"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307631523"}]},"ts":"1676307631523"} 2023-02-13 17:00:31,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:31,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:31,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e16e4423db3157675739f103f8f5a30, disabling compactions & flushes 2023-02-13 17:00:31,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:31,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:31,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. after waiting 0 ms 2023-02-13 17:00:31,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:31,525 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSED 2023-02-13 17:00:31,525 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631525"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307631525"}]},"ts":"1676307631525"} 2023-02-13 17:00:31,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:31,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:31,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7925c8f572739336270bf824c12623ea: Waiting for close lock at 1676307631519Running coprocessor pre-close hooks at 1676307631519Disabling compacts and flushes for region at 1676307631519Disabling writes for close at 1676307631520 (+1 ms)Writing region close event to WAL at 1676307631521 (+1 ms)Running coprocessor post-close hooks at 1676307631531 (+10 ms)Closed at 1676307631531 2023-02-13 17:00:31,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:31,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:31,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:31,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2457c5d248f99f39cff76beaa71b4345: Waiting for close lock at 1676307631521Running coprocessor pre-close hooks at 1676307631521Disabling compacts and flushes for region at 1676307631521Disabling writes for close at 1676307631522 (+1 ms)Writing region close event to WAL at 1676307631525 (+3 ms)Running coprocessor post-close hooks at 1676307631533 (+8 ms)Closed at 1676307631533 2023-02-13 17:00:31,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:31,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e16e4423db3157675739f103f8f5a30: Waiting for close lock at 1676307631524Running coprocessor pre-close hooks at 1676307631524Disabling compacts and flushes for region at 1676307631524Disabling writes for close at 1676307631524Writing region close event to WAL at 1676307631525 (+1 ms)Running coprocessor post-close hooks at 1676307631534 (+9 ms)Closed at 1676307631534 2023-02-13 17:00:31,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:31,538 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSED 2023-02-13 17:00:31,538 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631538"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307631538"}]},"ts":"1676307631538"} 2023-02-13 17:00:31,538 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:31,539 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=26 2023-02-13 17:00:31,539 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=25 2023-02-13 17:00:31,539 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=25, state=SUCCESS; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 182 msec 2023-02-13 17:00:31,539 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=26, state=SUCCESS; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747 in 186 msec 2023-02-13 17:00:31,539 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSED 2023-02-13 17:00:31,539 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307631539"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307631539"}]},"ts":"1676307631539"} 2023-02-13 17:00:31,540 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-02-13 17:00:31,540 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670 in 182 msec 2023-02-13 17:00:31,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:31,543 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN in 208 msec 2023-02-13 17:00:31,543 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN in 209 msec 2023-02-13 17:00:31,543 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN in 208 msec 2023-02-13 17:00:31,544 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSED 2023-02-13 17:00:31,544 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307631544"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307631544"}]},"ts":"1676307631544"} 2023-02-13 17:00:31,545 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=27 2023-02-13 17:00:31,545 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=27, state=SUCCESS; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716 in 185 msec 2023-02-13 17:00:31,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=24 2023-02-13 17:00:31,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=24, state=SUCCESS; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670 in 201 msec 2023-02-13 17:00:31,548 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN in 214 msec 2023-02-13 17:00:31,549 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN in 215 msec 2023-02-13 17:00:31,550 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=23 2023-02-13 17:00:31,550 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=23, state=SUCCESS; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 209 msec 2023-02-13 17:00:31,552 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-02-13 17:00:31,553 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN in 219 msec 2023-02-13 17:00:31,554 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307631553"}]},"ts":"1676307631553"} 2023-02-13 17:00:31,555 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLED in hbase:meta 2023-02-13 17:00:31,557 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676307630447 to state=DISABLED 2023-02-13 17:00:31,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676307630447 in 245 msec 2023-02-13 17:00:31,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 17:00:31,575 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 22 completed 2023-02-13 17:00:31,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:31,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676307631586 (current time:1676307631586). 2023-02-13 17:00:31,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 17:00:31,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(139): Connect 0x3d7b1050 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:31,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30d21f99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:31,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:31,600 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59380, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:31,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d7b1050 to 127.0.0.1:54083 2023-02-13 17:00:31,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:31,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 17:00:31,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:31,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:31,631 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:31,633 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE; timeout=600000, timestamp=1676308231633 2023-02-13 17:00:31,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:31,633 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676307630447 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 2023-02-13 17:00:31,639 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 17:00:31,640 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 17:00:31,664 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } 2023-02-13 17:00:31,669 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:31,669 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:31,669 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:31,669 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:31,670 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:31,670 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:31,672 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:31,676 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:31,676 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:31,676 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:31,676 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:31,676 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:31,676 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:31,676 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:31,686 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:31,686 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:31,686 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:31,687 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:31,687 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:31,688 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:31,704 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:31,890 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 17:00:31,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } is done 2023-02-13 17:00:31,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 17:00:31,894 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 17:00:32,188 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } as finished. 2023-02-13 17:00:32,188 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676307630447 2023-02-13 17:00:32,191 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/emptySnaptb-1676307630447 2023-02-13 17:00:32,243 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/emptySnaptb-1676307630447 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/emptySnaptb-1676307630447 2023-02-13 17:00:32,277 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676307630447 of table testRestoreSnapshot-1676307630447 completed 2023-02-13 17:00:32,277 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/emptySnaptb-1676307630447 2023-02-13 17:00:32,278 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/emptySnaptb-1676307630447 2023-02-13 17:00:32,278 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676307630447 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 at 1676307631633Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } at 1676307631664 (+31 ms)Consolidate snapshot: emptySnaptb-1676307630447 at 1676307632188 (+524 ms)Loading Region manifests for emptySnaptb-1676307630447 at 1676307632189 (+1 ms)Writing data manifest for emptySnaptb-1676307630447 at 1676307632203 (+14 ms)Verifying snapshot: emptySnaptb-1676307630447 at 1676307632227 (+24 ms)Snapshot emptySnaptb-1676307630447 of table testRestoreSnapshot-1676307630447 completed at 1676307632277 (+50 ms) 2023-02-13 17:00:32,279 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:32,281 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE in 659 msec 2023-02-13 17:00:32,395 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 17:00:32,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } is done 2023-02-13 17:00:32,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 17:00:32,398 INFO [Listener at localhost/40167] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:32,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676307630447 2023-02-13 17:00:32,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:32,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 17:00:32,413 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676307630447 2023-02-13 17:00:32,413 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307632413"}]},"ts":"1676307632413"} 2023-02-13 17:00:32,415 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLING in hbase:meta 2023-02-13 17:00:32,419 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 17:00:32,421 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN}] 2023-02-13 17:00:32,423 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN 2023-02-13 17:00:32,423 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN 2023-02-13 17:00:32,423 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN 2023-02-13 17:00:32,423 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN 2023-02-13 17:00:32,424 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:32,424 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=true 2023-02-13 17:00:32,425 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN 2023-02-13 17:00:32,425 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:32,425 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=true 2023-02-13 17:00:32,425 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:32,425 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:32,425 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN 2023-02-13 17:00:32,426 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=true 2023-02-13 17:00:32,425 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=true 2023-02-13 17:00:32,427 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:32,427 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=true 2023-02-13 17:00:32,427 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:32,427 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=true 2023-02-13 17:00:32,575 INFO [jenkins-hbase4:40255] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 17:00:32,575 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:32,575 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:32,575 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:32,575 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:32,575 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307632575"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307632575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307632575"}]},"ts":"1676307632575"} 2023-02-13 17:00:32,575 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:32,576 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307632575"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307632575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307632575"}]},"ts":"1676307632575"} 2023-02-13 17:00:32,576 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307632575"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307632575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307632575"}]},"ts":"1676307632575"} 2023-02-13 17:00:32,575 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307632575"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307632575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307632575"}]},"ts":"1676307632575"} 2023-02-13 17:00:32,575 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307632575"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307632575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307632575"}]},"ts":"1676307632575"} 2023-02-13 17:00:32,578 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=41, state=RUNNABLE; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:32,579 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=37, state=RUNNABLE; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:32,581 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=42, state=RUNNABLE; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:32,582 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=38, state=RUNNABLE; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:32,583 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:32,583 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=39, state=RUNNABLE; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:32,583 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307632583"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307632583"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307632583"}]},"ts":"1676307632583"} 2023-02-13 17:00:32,586 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=40, state=RUNNABLE; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:32,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 17:00:32,736 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:32,736 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:32,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 17:00:32,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 17:00:32,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:32,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:32,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:32,737 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:32,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:32,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:32,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 17:00:32,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:32,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:32,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:32,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:32,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:32,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:32,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:32,739 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:32,739 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:32,739 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:32,741 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:32,741 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:32,741 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:32,741 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:32,741 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:32,741 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:32,742 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f31ed543f552e93833ffd6a17e17334b columnFamilyName cf 2023-02-13 17:00:32,743 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61523f42895e23c1110bbb6a6531b0a6 columnFamilyName cf 2023-02-13 17:00:32,743 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7925c8f572739336270bf824c12623ea columnFamilyName cf 2023-02-13 17:00:32,743 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(310): Store=f31ed543f552e93833ffd6a17e17334b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:32,744 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(310): Store=61523f42895e23c1110bbb6a6531b0a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:32,745 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(310): Store=7925c8f572739336270bf824c12623ea/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:32,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:32,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:32,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:32,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:32,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:32,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:32,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:32,752 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:32,752 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:32,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f31ed543f552e93833ffd6a17e17334b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74866837, jitterRate=0.11560280621051788}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:32,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f31ed543f552e93833ffd6a17e17334b: Running coprocessor pre-open hook at 1676307632738Writing region info on filesystem at 1676307632738Initializing all the Stores at 1676307632739 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307632739Cleaning up temporary data from old regions at 1676307632747 (+8 ms)Cleaning up detritus from prior splits at 1676307632748 (+1 ms)Running coprocessor post-open hooks at 1676307632753 (+5 ms)Region opened successfully at 1676307632753 2023-02-13 17:00:32,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 61523f42895e23c1110bbb6a6531b0a6; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63278538, jitterRate=-0.05707630515098572}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:32,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 61523f42895e23c1110bbb6a6531b0a6: Running coprocessor pre-open hook at 1676307632738Writing region info on filesystem at 1676307632738Initializing all the Stores at 1676307632739 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307632739Cleaning up temporary data from old regions at 1676307632748 (+9 ms)Cleaning up detritus from prior splits at 1676307632749 (+1 ms)Running coprocessor post-open hooks at 1676307632753 (+4 ms)Region opened successfully at 1676307632753 2023-02-13 17:00:32,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6., pid=45, masterSystemTime=1676307632733 2023-02-13 17:00:32,756 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7925c8f572739336270bf824c12623ea; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73373802, jitterRate=0.09335485100746155}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:32,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7925c8f572739336270bf824c12623ea: Running coprocessor pre-open hook at 1676307632738Writing region info on filesystem at 1676307632738Initializing all the Stores at 1676307632739 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307632739Cleaning up temporary data from old regions at 1676307632749 (+10 ms)Cleaning up detritus from prior splits at 1676307632749Running coprocessor post-open hooks at 1676307632756 (+7 ms)Region opened successfully at 1676307632756 2023-02-13 17:00:32,756 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b., pid=48, masterSystemTime=1676307632732 2023-02-13 17:00:32,760 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea., pid=43, masterSystemTime=1676307632731 2023-02-13 17:00:32,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:32,761 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:32,761 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:32,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 17:00:32,762 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:32,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:32,762 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307632762"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307632762"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307632762"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307632762"}]},"ts":"1676307632762"} 2023-02-13 17:00:32,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:32,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:32,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:32,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 17:00:32,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:32,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:32,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:32,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:32,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:32,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:32,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:32,764 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:32,765 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307632764"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307632764"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307632764"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307632764"}]},"ts":"1676307632764"} 2023-02-13 17:00:32,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:32,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:32,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:32,766 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:32,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 17:00:32,766 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307632766"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307632766"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307632766"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307632766"}]},"ts":"1676307632766"} 2023-02-13 17:00:32,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:32,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:32,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:32,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:32,767 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:32,768 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:32,769 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:32,769 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:32,770 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:32,770 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2457c5d248f99f39cff76beaa71b4345 columnFamilyName cf 2023-02-13 17:00:32,770 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:32,771 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e16e4423db3157675739f103f8f5a30 columnFamilyName cf 2023-02-13 17:00:32,771 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=42 2023-02-13 17:00:32,771 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=42, state=SUCCESS; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670 in 185 msec 2023-02-13 17:00:32,773 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=40 2023-02-13 17:00:32,773 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=40, state=SUCCESS; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747 in 182 msec 2023-02-13 17:00:32,773 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(310): Store=0e16e4423db3157675739f103f8f5a30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:32,774 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(310): Store=2457c5d248f99f39cff76beaa71b4345/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:32,775 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN in 352 msec 2023-02-13 17:00:32,775 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=41 2023-02-13 17:00:32,775 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=41, state=SUCCESS; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716 in 193 msec 2023-02-13 17:00:32,776 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN in 354 msec 2023-02-13 17:00:32,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN in 356 msec 2023-02-13 17:00:32,778 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:32,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:32,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:32,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:32,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:32,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:32,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:32,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2457c5d248f99f39cff76beaa71b4345; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72503803, jitterRate=0.08039085566997528}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:32,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2457c5d248f99f39cff76beaa71b4345: Running coprocessor pre-open hook at 1676307632764Writing region info on filesystem at 1676307632764Initializing all the Stores at 1676307632766 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307632766Cleaning up temporary data from old regions at 1676307632782 (+16 ms)Cleaning up detritus from prior splits at 1676307632783 (+1 ms)Running coprocessor post-open hooks at 1676307632788 (+5 ms)Region opened successfully at 1676307632788 2023-02-13 17:00:32,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e16e4423db3157675739f103f8f5a30; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69329763, jitterRate=0.033093973994255066}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:32,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e16e4423db3157675739f103f8f5a30: Running coprocessor pre-open hook at 1676307632763Writing region info on filesystem at 1676307632763Initializing all the Stores at 1676307632766 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307632766Cleaning up temporary data from old regions at 1676307632782 (+16 ms)Cleaning up detritus from prior splits at 1676307632783 (+1 ms)Running coprocessor post-open hooks at 1676307632788 (+5 ms)Region opened successfully at 1676307632788 2023-02-13 17:00:32,789 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:32,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30., pid=44, masterSystemTime=1676307632732 2023-02-13 17:00:32,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345., pid=46, masterSystemTime=1676307632733 2023-02-13 17:00:32,789 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:32,790 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 columnFamilyName cf 2023-02-13 17:00:32,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:32,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:32,793 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:32,794 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307632793"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307632793"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307632793"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307632793"}]},"ts":"1676307632793"} 2023-02-13 17:00:32,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:32,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:32,795 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(310): Store=bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:32,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:32,797 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:32,797 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307632797"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307632797"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307632797"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307632797"}]},"ts":"1676307632797"} 2023-02-13 17:00:32,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:32,803 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=37 2023-02-13 17:00:32,803 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=37, state=SUCCESS; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 218 msec 2023-02-13 17:00:32,804 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=38 2023-02-13 17:00:32,804 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=38, state=SUCCESS; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670 in 218 msec 2023-02-13 17:00:32,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:32,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN in 384 msec 2023-02-13 17:00:32,806 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN in 385 msec 2023-02-13 17:00:32,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bb0e3d66ca9f4f9ce3e87dc8bfc74c30; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66498562, jitterRate=-0.009094208478927612}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:32,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Running coprocessor pre-open hook at 1676307632767Writing region info on filesystem at 1676307632767Initializing all the Stores at 1676307632769 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307632769Cleaning up temporary data from old regions at 1676307632800 (+31 ms)Cleaning up detritus from prior splits at 1676307632801 (+1 ms)Running coprocessor post-open hooks at 1676307632807 (+6 ms)Region opened successfully at 1676307632807 2023-02-13 17:00:32,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30., pid=47, masterSystemTime=1676307632731 2023-02-13 17:00:32,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:32,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:32,810 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:32,811 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307632810"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307632810"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307632810"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307632810"}]},"ts":"1676307632810"} 2023-02-13 17:00:32,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=39 2023-02-13 17:00:32,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=39, state=SUCCESS; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 230 msec 2023-02-13 17:00:32,818 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=36 2023-02-13 17:00:32,818 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN in 397 msec 2023-02-13 17:00:32,819 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307632819"}]},"ts":"1676307632819"} 2023-02-13 17:00:32,821 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLED in hbase:meta 2023-02-13 17:00:32,823 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676307630447' was successfully enabled. 2023-02-13 17:00:32,827 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676307630447 in 421 msec 2023-02-13 17:00:33,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 17:00:33,167 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 36 completed 2023-02-13 17:00:33,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:33,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:33,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:33,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:33,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:33,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:33,226 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676307630447 2023-02-13 17:00:33,226 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:33,227 INFO [Listener at localhost/40167] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:00:33,247 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 17:00:33,249 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 17:00:33,251 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 17:00:33,252 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 17:00:33,255 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 17:00:33,265 INFO [Listener at localhost/40167] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:33,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676307630447 2023-02-13 17:00:33,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:33,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 17:00:33,271 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307633271"}]},"ts":"1676307633271"} 2023-02-13 17:00:33,273 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLING in hbase:meta 2023-02-13 17:00:33,275 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676307630447 to state=DISABLING 2023-02-13 17:00:33,275 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN}] 2023-02-13 17:00:33,278 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN 2023-02-13 17:00:33,278 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN 2023-02-13 17:00:33,278 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN 2023-02-13 17:00:33,278 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN 2023-02-13 17:00:33,278 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN 2023-02-13 17:00:33,279 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:33,279 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:33,279 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:33,279 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307633279"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307633279"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307633279"}]},"ts":"1676307633279"} 2023-02-13 17:00:33,279 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:33,279 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307633279"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307633279"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307633279"}]},"ts":"1676307633279"} 2023-02-13 17:00:33,280 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307633279"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307633279"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307633279"}]},"ts":"1676307633279"} 2023-02-13 17:00:33,279 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307633279"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307633279"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307633279"}]},"ts":"1676307633279"} 2023-02-13 17:00:33,280 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:33,280 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307633280"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307633280"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307633280"}]},"ts":"1676307633280"} 2023-02-13 17:00:33,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=50, state=RUNNABLE; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:33,284 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=53, state=RUNNABLE; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:33,285 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=52, state=RUNNABLE; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:33,286 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=51, state=RUNNABLE; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:33,287 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=55, state=RUNNABLE; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:33,290 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN 2023-02-13 17:00:33,291 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:33,292 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307633291"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307633291"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307633291"}]},"ts":"1676307633291"} 2023-02-13 17:00:33,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=54, state=RUNNABLE; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:33,437 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:33,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e16e4423db3157675739f103f8f5a30, disabling compactions & flushes 2023-02-13 17:00:33,437 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:33,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:33,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. after waiting 0 ms 2023-02-13 17:00:33,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:33,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 0e16e4423db3157675739f103f8f5a30 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-13 17:00:33,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:33,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7925c8f572739336270bf824c12623ea, disabling compactions & flushes 2023-02-13 17:00:33,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:33,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:33,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. after waiting 0 ms 2023-02-13 17:00:33,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:33,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7925c8f572739336270bf824c12623ea 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-13 17:00:33,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:33,441 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2457c5d248f99f39cff76beaa71b4345, disabling compactions & flushes 2023-02-13 17:00:33,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:33,441 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:33,441 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. after waiting 0 ms 2023-02-13 17:00:33,441 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:33,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 2457c5d248f99f39cff76beaa71b4345 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-13 17:00:33,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 17:00:33,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 2023-02-13 17:00:33,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 2023-02-13 17:00:33,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 2023-02-13 17:00:33,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/.tmp/cf/15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:33,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/.tmp/cf/52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:33,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/.tmp/cf/47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:33,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/.tmp/cf/47c584031cad45b68b3da5b7c3330a2d as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:33,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/.tmp/cf/15eaf8061d754363820c5e36fdb4956a as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:33,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/.tmp/cf/52130edd005d4ce89422019dd11d135a as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:33,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d, entries=31, sequenceid=8, filesize=10.2 K 2023-02-13 17:00:33,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a, entries=33, sequenceid=8, filesize=10.5 K 2023-02-13 17:00:33,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a, entries=36, sequenceid=8, filesize=11.0 K 2023-02-13 17:00:33,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 0e16e4423db3157675739f103f8f5a30 in 182ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:33,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 2457c5d248f99f39cff76beaa71b4345 in 179ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:33,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 7925c8f572739336270bf824c12623ea in 180ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:33,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676307630447' 2023-02-13 17:00:33,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676307630447' 2023-02-13 17:00:33,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676307630447' 2023-02-13 17:00:33,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:33,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:33,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:33,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:33,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:33,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:33,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7925c8f572739336270bf824c12623ea: Waiting for close lock at 1676307633439Running coprocessor pre-close hooks at 1676307633439Disabling compacts and flushes for region at 1676307633439Disabling writes for close at 1676307633440 (+1 ms)Obtaining lock to block concurrent updates at 1676307633440Preparing flush snapshotting stores in 7925c8f572739336270bf824c12623ea at 1676307633440Finished memstore snapshotting testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676307633449 (+9 ms)Flushing stores of testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. at 1676307633449Flushing 7925c8f572739336270bf824c12623ea/cf: creating writer at 1676307633452 (+3 ms)Flushing mob file 7925c8f572739336270bf824c12623ea/cf: appending metadata at 1676307633490 (+38 ms)Flushing mob file 7925c8f572739336270bf824c12623ea/cf: closing flushed file at 1676307633490Flushing 7925c8f572739336270bf824c12623ea/cf: appending metadata at 1676307633561 (+71 ms)Flushing 7925c8f572739336270bf824c12623ea/cf: closing flushed file at 1676307633565 (+4 ms)Flushing 7925c8f572739336270bf824c12623ea/cf: reopening flushed file at 1676307633609 (+44 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 7925c8f572739336270bf824c12623ea in 180ms, sequenceid=8, compaction requested=false at 1676307633620 (+11 ms)Writing region close event to WAL at 1676307633627 (+7 ms)Running coprocessor post-close hooks at 1676307633635 (+8 ms)Closed at 1676307633635 2023-02-13 17:00:33,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2457c5d248f99f39cff76beaa71b4345: Waiting for close lock at 1676307633440Running coprocessor pre-close hooks at 1676307633440Disabling compacts and flushes for region at 1676307633440Disabling writes for close at 1676307633441 (+1 ms)Obtaining lock to block concurrent updates at 1676307633441Preparing flush snapshotting stores in 2457c5d248f99f39cff76beaa71b4345 at 1676307633441Finished memstore snapshotting testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676307633449 (+8 ms)Flushing stores of testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. at 1676307633449Flushing 2457c5d248f99f39cff76beaa71b4345/cf: creating writer at 1676307633452 (+3 ms)Flushing mob file 2457c5d248f99f39cff76beaa71b4345/cf: appending metadata at 1676307633490 (+38 ms)Flushing mob file 2457c5d248f99f39cff76beaa71b4345/cf: closing flushed file at 1676307633490Flushing 2457c5d248f99f39cff76beaa71b4345/cf: appending metadata at 1676307633562 (+72 ms)Flushing 2457c5d248f99f39cff76beaa71b4345/cf: closing flushed file at 1676307633565 (+3 ms)Flushing 2457c5d248f99f39cff76beaa71b4345/cf: reopening flushed file at 1676307633609 (+44 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 2457c5d248f99f39cff76beaa71b4345 in 179ms, sequenceid=8, compaction requested=false at 1676307633620 (+11 ms)Writing region close event to WAL at 1676307633627 (+7 ms)Running coprocessor post-close hooks at 1676307633635 (+8 ms)Closed at 1676307633635 2023-02-13 17:00:33,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e16e4423db3157675739f103f8f5a30: Waiting for close lock at 1676307633437Running coprocessor pre-close hooks at 1676307633437Disabling compacts and flushes for region at 1676307633437Disabling writes for close at 1676307633437Obtaining lock to block concurrent updates at 1676307633438 (+1 ms)Preparing flush snapshotting stores in 0e16e4423db3157675739f103f8f5a30 at 1676307633438Finished memstore snapshotting testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676307633449 (+11 ms)Flushing stores of testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. at 1676307633449Flushing 0e16e4423db3157675739f103f8f5a30/cf: creating writer at 1676307633452 (+3 ms)Flushing mob file 0e16e4423db3157675739f103f8f5a30/cf: appending metadata at 1676307633490 (+38 ms)Flushing mob file 0e16e4423db3157675739f103f8f5a30/cf: closing flushed file at 1676307633490Flushing 0e16e4423db3157675739f103f8f5a30/cf: appending metadata at 1676307633562 (+72 ms)Flushing 0e16e4423db3157675739f103f8f5a30/cf: closing flushed file at 1676307633565 (+3 ms)Flushing 0e16e4423db3157675739f103f8f5a30/cf: reopening flushed file at 1676307633609 (+44 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 0e16e4423db3157675739f103f8f5a30 in 182ms, sequenceid=8, compaction requested=false at 1676307633620 (+11 ms)Writing region close event to WAL at 1676307633627 (+7 ms)Running coprocessor post-close hooks at 1676307633635 (+8 ms)Closed at 1676307633635 2023-02-13 17:00:33,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:33,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:33,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f31ed543f552e93833ffd6a17e17334b, disabling compactions & flushes 2023-02-13 17:00:33,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:33,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:33,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. after waiting 0 ms 2023-02-13 17:00:33,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:33,638 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f31ed543f552e93833ffd6a17e17334b 1/1 column families, dataSize=1.50 KB heapSize=3.48 KB 2023-02-13 17:00:33,638 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSED 2023-02-13 17:00:33,638 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307633638"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307633638"}]},"ts":"1676307633638"} 2023-02-13 17:00:33,638 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:33,638 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:33,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bb0e3d66ca9f4f9ce3e87dc8bfc74c30, disabling compactions & flushes 2023-02-13 17:00:33,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:33,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:33,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. after waiting 0 ms 2023-02-13 17:00:33,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:33,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing bb0e3d66ca9f4f9ce3e87dc8bfc74c30 1/1 column families, dataSize=1.50 KB heapSize=3.48 KB 2023-02-13 17:00:33,641 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSED 2023-02-13 17:00:33,641 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307633641"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307633641"}]},"ts":"1676307633641"} 2023-02-13 17:00:33,641 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:33,641 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:33,642 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 61523f42895e23c1110bbb6a6531b0a6, disabling compactions & flushes 2023-02-13 17:00:33,642 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:33,642 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:33,642 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. after waiting 0 ms 2023-02-13 17:00:33,642 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:33,643 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 61523f42895e23c1110bbb6a6531b0a6 1/1 column families, dataSize=23.16 KB heapSize=50.03 KB 2023-02-13 17:00:33,643 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSED 2023-02-13 17:00:33,644 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307633643"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307633643"}]},"ts":"1676307633643"} 2023-02-13 17:00:33,648 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=50 2023-02-13 17:00:33,648 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=50, state=SUCCESS; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 362 msec 2023-02-13 17:00:33,651 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN in 373 msec 2023-02-13 17:00:33,653 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=51 2023-02-13 17:00:33,653 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=54 2023-02-13 17:00:33,653 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=51, state=SUCCESS; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670 in 364 msec 2023-02-13 17:00:33,653 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=54, state=SUCCESS; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716 in 356 msec 2023-02-13 17:00:33,656 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN in 378 msec 2023-02-13 17:00:33,656 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN in 378 msec 2023-02-13 17:00:33,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 2023-02-13 17:00:33,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 2023-02-13 17:00:33,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/.tmp/cf/1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:33,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/.tmp/cf/97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:33,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/.tmp/cf/1b81c71679994dda95fb56406c0c1e6d as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:33,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/.tmp/cf/97a455c0271c489d81c6c7b49b2cf4d9 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:33,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d, entries=23, sequenceid=8, filesize=8.8 K 2023-02-13 17:00:33,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for f31ed543f552e93833ffd6a17e17334b in 82ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:33,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9, entries=23, sequenceid=8, filesize=8.8 K 2023-02-13 17:00:33,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 in 85ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:33,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:33,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:33,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f31ed543f552e93833ffd6a17e17334b: Waiting for close lock at 1676307633637Running coprocessor pre-close hooks at 1676307633637Disabling compacts and flushes for region at 1676307633637Disabling writes for close at 1676307633637Obtaining lock to block concurrent updates at 1676307633638 (+1 ms)Preparing flush snapshotting stores in f31ed543f552e93833ffd6a17e17334b at 1676307633638Finished memstore snapshotting testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b., syncing WAL and waiting on mvcc, flushsize=dataSize=1539, getHeapSize=3552, getOffHeapSize=0, getCellsCount=23 at 1676307633638Flushing stores of testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. at 1676307633640 (+2 ms)Flushing f31ed543f552e93833ffd6a17e17334b/cf: creating writer at 1676307633641 (+1 ms)Flushing mob file f31ed543f552e93833ffd6a17e17334b/cf: appending metadata at 1676307633650 (+9 ms)Flushing mob file f31ed543f552e93833ffd6a17e17334b/cf: closing flushed file at 1676307633650Flushing f31ed543f552e93833ffd6a17e17334b/cf: appending metadata at 1676307633682 (+32 ms)Flushing f31ed543f552e93833ffd6a17e17334b/cf: closing flushed file at 1676307633682Flushing f31ed543f552e93833ffd6a17e17334b/cf: reopening flushed file at 1676307633712 (+30 ms)Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for f31ed543f552e93833ffd6a17e17334b in 82ms, sequenceid=8, compaction requested=false at 1676307633720 (+8 ms)Writing region close event to WAL at 1676307633722 (+2 ms)Running coprocessor post-close hooks at 1676307633728 (+6 ms)Closed at 1676307633728 2023-02-13 17:00:33,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:33,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:33,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Waiting for close lock at 1676307633638Running coprocessor pre-close hooks at 1676307633638Disabling compacts and flushes for region at 1676307633638Disabling writes for close at 1676307633639 (+1 ms)Obtaining lock to block concurrent updates at 1676307633639Preparing flush snapshotting stores in bb0e3d66ca9f4f9ce3e87dc8bfc74c30 at 1676307633639Finished memstore snapshotting testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30., syncing WAL and waiting on mvcc, flushsize=dataSize=1539, getHeapSize=3552, getOffHeapSize=0, getCellsCount=23 at 1676307633640 (+1 ms)Flushing stores of testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. at 1676307633643 (+3 ms)Flushing bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: creating writer at 1676307633643Flushing mob file bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: appending metadata at 1676307633656 (+13 ms)Flushing mob file bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: closing flushed file at 1676307633656Flushing bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: appending metadata at 1676307633684 (+28 ms)Flushing bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: closing flushed file at 1676307633684Flushing bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: reopening flushed file at 1676307633714 (+30 ms)Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 in 85ms, sequenceid=8, compaction requested=false at 1676307633724 (+10 ms)Writing region close event to WAL at 1676307633726 (+2 ms)Running coprocessor post-close hooks at 1676307633730 (+4 ms)Closed at 1676307633730 2023-02-13 17:00:33,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:33,732 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSED 2023-02-13 17:00:33,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:33,732 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307633732"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307633732"}]},"ts":"1676307633732"} 2023-02-13 17:00:33,732 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSED 2023-02-13 17:00:33,732 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307633732"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307633732"}]},"ts":"1676307633732"} 2023-02-13 17:00:33,736 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=53 2023-02-13 17:00:33,736 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=53, state=SUCCESS; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747 in 450 msec 2023-02-13 17:00:33,737 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=52 2023-02-13 17:00:33,737 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN in 461 msec 2023-02-13 17:00:33,738 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=52, state=SUCCESS; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 449 msec 2023-02-13 17:00:33,739 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN in 462 msec 2023-02-13 17:00:34,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 17:00:34,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 2023-02-13 17:00:34,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=23.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/.tmp/cf/522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:34,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/.tmp/cf/522b6eff1a16428ab53f7cfcc13f5481 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:34,129 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481, entries=354, sequenceid=8, filesize=65.2 K 2023-02-13 17:00:34,130 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.16 KB/23714, heapSize ~50.02 KB/51216, currentSize=0 B/0 for 61523f42895e23c1110bbb6a6531b0a6 in 488ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:34,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:34,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:34,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 61523f42895e23c1110bbb6a6531b0a6: Waiting for close lock at 1676307633641Running coprocessor pre-close hooks at 1676307633641Disabling compacts and flushes for region at 1676307633641Disabling writes for close at 1676307633642 (+1 ms)Obtaining lock to block concurrent updates at 1676307633643 (+1 ms)Preparing flush snapshotting stores in 61523f42895e23c1110bbb6a6531b0a6 at 1676307633643Finished memstore snapshotting testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6., syncing WAL and waiting on mvcc, flushsize=dataSize=23714, getHeapSize=51216, getOffHeapSize=0, getCellsCount=354 at 1676307633643Flushing stores of testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. at 1676307633646 (+3 ms)Flushing 61523f42895e23c1110bbb6a6531b0a6/cf: creating writer at 1676307633648 (+2 ms)Flushing mob file 61523f42895e23c1110bbb6a6531b0a6/cf: appending metadata at 1676307633673 (+25 ms)Flushing mob file 61523f42895e23c1110bbb6a6531b0a6/cf: closing flushed file at 1676307633673Flushing 61523f42895e23c1110bbb6a6531b0a6/cf: appending metadata at 1676307634100 (+427 ms)Flushing 61523f42895e23c1110bbb6a6531b0a6/cf: closing flushed file at 1676307634101 (+1 ms)Flushing 61523f42895e23c1110bbb6a6531b0a6/cf: reopening flushed file at 1676307634124 (+23 ms)Finished flush of dataSize ~23.16 KB/23714, heapSize ~50.02 KB/51216, currentSize=0 B/0 for 61523f42895e23c1110bbb6a6531b0a6 in 488ms, sequenceid=8, compaction requested=false at 1676307634130 (+6 ms)Writing region close event to WAL at 1676307634132 (+2 ms)Running coprocessor post-close hooks at 1676307634136 (+4 ms)Closed at 1676307634136 2023-02-13 17:00:34,138 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:34,139 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSED 2023-02-13 17:00:34,139 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307634139"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307634139"}]},"ts":"1676307634139"} 2023-02-13 17:00:34,143 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=55 2023-02-13 17:00:34,143 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=55, state=SUCCESS; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670 in 854 msec 2023-02-13 17:00:34,145 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=49 2023-02-13 17:00:34,145 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN in 868 msec 2023-02-13 17:00:34,146 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307634146"}]},"ts":"1676307634146"} 2023-02-13 17:00:34,148 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLED in hbase:meta 2023-02-13 17:00:34,150 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676307630447 to state=DISABLED 2023-02-13 17:00:34,152 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676307630447 in 885 msec 2023-02-13 17:00:34,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 17:00:34,776 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 49 completed 2023-02-13 17:00:34,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:34,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676307634777 (current time:1676307634777). 2023-02-13 17:00:34,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 17:00:34,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(139): Connect 0x4d061a01 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:34,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56f87ec6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:34,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:34,788 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34738, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:34,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d061a01 to 127.0.0.1:54083 2023-02-13 17:00:34,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:34,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 17:00:34,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:34,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:34,794 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:34,795 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE; timeout=600000, timestamp=1676308234795 2023-02-13 17:00:34,795 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676307630447 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 2023-02-13 17:00:34,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:34,795 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 17:00:34,795 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 17:00:34,813 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } 2023-02-13 17:00:34,814 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:34,814 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:34,815 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:34,815 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:34,817 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:34,817 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:34,818 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:34,818 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:34,819 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:34,819 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:34,819 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:34,819 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:34,819 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:34,819 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:34,822 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a] hfiles 2023-02-13 17:00:34,822 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a] hfiles 2023-02-13 17:00:34,822 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:34,823 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:34,827 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1] hfiles 2023-02-13 17:00:34,827 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 2023-02-13 17:00:34,828 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 2023-02-13 17:00:34,828 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 2023-02-13 17:00:34,828 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 2023-02-13 17:00:34,828 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 2023-02-13 17:00:34,828 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 2023-02-13 17:00:34,831 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9] hfiles 2023-02-13 17:00:34,831 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:34,831 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d] hfiles 2023-02-13 17:00:34,832 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d] hfiles 2023-02-13 17:00:34,832 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481] hfiles 2023-02-13 17:00:34,832 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:34,832 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:34,832 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:34,958 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 17:00:35,026 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 17:00:35,027 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 17:00:35,045 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 17:00:35,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } is done 2023-02-13 17:00:35,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 17:00:35,047 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 17:00:35,261 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } as finished. 2023-02-13 17:00:35,261 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676307630447 2023-02-13 17:00:35,263 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb0-1676307630447 2023-02-13 17:00:35,329 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb0-1676307630447 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/snaptb0-1676307630447 2023-02-13 17:00:35,363 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676307630447 of table testRestoreSnapshot-1676307630447 completed 2023-02-13 17:00:35,363 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb0-1676307630447 2023-02-13 17:00:35,363 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb0-1676307630447 2023-02-13 17:00:35,363 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676307630447 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 at 1676307634795Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } at 1676307634813 (+18 ms)Consolidate snapshot: snaptb0-1676307630447 at 1676307635261 (+448 ms)Loading Region manifests for snaptb0-1676307630447 at 1676307635262 (+1 ms)Writing data manifest for snaptb0-1676307630447 at 1676307635284 (+22 ms)Verifying snapshot: snaptb0-1676307630447 at 1676307635305 (+21 ms)Snapshot snaptb0-1676307630447 of table testRestoreSnapshot-1676307630447 completed at 1676307635363 (+58 ms) 2023-02-13 17:00:35,365 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:35,367 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE in 573 msec 2023-02-13 17:00:35,547 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 17:00:35,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } is done 2023-02-13 17:00:35,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 17:00:35,549 INFO [Listener at localhost/40167] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:35,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676307630447 2023-02-13 17:00:35,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:35,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 17:00:35,554 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676307630447 2023-02-13 17:00:35,554 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307635554"}]},"ts":"1676307635554"} 2023-02-13 17:00:35,555 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLING in hbase:meta 2023-02-13 17:00:35,561 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 17:00:35,561 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN}] 2023-02-13 17:00:35,563 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN 2023-02-13 17:00:35,563 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN 2023-02-13 17:00:35,563 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN 2023-02-13 17:00:35,563 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN 2023-02-13 17:00:35,564 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN 2023-02-13 17:00:35,564 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:35,564 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:35,564 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:35,564 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=true 2023-02-13 17:00:35,564 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=true 2023-02-13 17:00:35,564 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=true 2023-02-13 17:00:35,564 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:35,565 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=true 2023-02-13 17:00:35,565 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:35,565 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=true 2023-02-13 17:00:35,565 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN 2023-02-13 17:00:35,568 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:35,568 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=true 2023-02-13 17:00:35,715 INFO [jenkins-hbase4:40255] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 17:00:35,715 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:35,715 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:35,715 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:35,715 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:35,715 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307635715"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307635715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307635715"}]},"ts":"1676307635715"} 2023-02-13 17:00:35,715 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307635715"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307635715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307635715"}]},"ts":"1676307635715"} 2023-02-13 17:00:35,715 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:35,715 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307635715"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307635715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307635715"}]},"ts":"1676307635715"} 2023-02-13 17:00:35,715 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307635715"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307635715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307635715"}]},"ts":"1676307635715"} 2023-02-13 17:00:35,716 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307635715"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307635715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307635715"}]},"ts":"1676307635715"} 2023-02-13 17:00:35,718 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=66, state=RUNNABLE; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:35,719 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=68, state=RUNNABLE; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:35,720 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=64, state=RUNNABLE; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:35,721 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=69, state=RUNNABLE; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:35,722 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:35,722 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307635722"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307635722"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307635722"}]},"ts":"1676307635722"} 2023-02-13 17:00:35,723 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=65, state=RUNNABLE; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:35,728 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=67, state=RUNNABLE; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:35,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 17:00:35,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:35,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 17:00:35,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:35,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:35,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:35,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:35,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:35,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 17:00:35,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:35,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:35,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:35,876 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:35,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:35,878 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:35,878 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:35,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:35,878 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:35,878 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 columnFamilyName cf 2023-02-13 17:00:35,878 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 17:00:35,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:35,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:35,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:35,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:35,879 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:35,879 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:35,880 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e16e4423db3157675739f103f8f5a30 columnFamilyName cf 2023-02-13 17:00:35,884 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:35,886 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:35,886 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:35,886 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2457c5d248f99f39cff76beaa71b4345 columnFamilyName cf 2023-02-13 17:00:35,892 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:35,892 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:35,893 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(310): Store=0e16e4423db3157675739f103f8f5a30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:35,893 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(310): Store=bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:35,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:35,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:35,896 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:35,897 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(310): Store=2457c5d248f99f39cff76beaa71b4345/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:35,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:35,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:35,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:35,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:35,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:35,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:35,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e16e4423db3157675739f103f8f5a30; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69620814, jitterRate=0.03743097186088562}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:35,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e16e4423db3157675739f103f8f5a30: Running coprocessor pre-open hook at 1676307635877Writing region info on filesystem at 1676307635877Initializing all the Stores at 1676307635878 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307635878Cleaning up temporary data from old regions at 1676307635898 (+20 ms)Cleaning up detritus from prior splits at 1676307635898Running coprocessor post-open hooks at 1676307635902 (+4 ms)Region opened successfully at 1676307635902 2023-02-13 17:00:35,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bb0e3d66ca9f4f9ce3e87dc8bfc74c30; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67977694, jitterRate=0.01294657588005066}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:35,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Running coprocessor pre-open hook at 1676307635875Writing region info on filesystem at 1676307635875Initializing all the Stores at 1676307635876 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307635876Cleaning up temporary data from old regions at 1676307635898 (+22 ms)Cleaning up detritus from prior splits at 1676307635899 (+1 ms)Running coprocessor post-open hooks at 1676307635903 (+4 ms)Region opened successfully at 1676307635903 2023-02-13 17:00:35,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30., pid=70, masterSystemTime=1676307635870 2023-02-13 17:00:35,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30., pid=72, masterSystemTime=1676307635872 2023-02-13 17:00:35,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:35,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:35,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:35,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 17:00:35,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:35,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:35,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:35,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:35,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:35,907 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:35,908 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307635907"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307635907"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307635907"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307635907"}]},"ts":"1676307635907"} 2023-02-13 17:00:35,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:35,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:35,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:35,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 17:00:35,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2457c5d248f99f39cff76beaa71b4345; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60574482, jitterRate=-0.09736987948417664}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:35,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2457c5d248f99f39cff76beaa71b4345: Running coprocessor pre-open hook at 1676307635879Writing region info on filesystem at 1676307635879Initializing all the Stores at 1676307635884 (+5 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307635884Cleaning up temporary data from old regions at 1676307635900 (+16 ms)Cleaning up detritus from prior splits at 1676307635904 (+4 ms)Running coprocessor post-open hooks at 1676307635909 (+5 ms)Region opened successfully at 1676307635909 2023-02-13 17:00:35,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:35,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:35,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:35,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:35,910 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:35,910 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307635910"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307635910"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307635910"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307635910"}]},"ts":"1676307635910"} 2023-02-13 17:00:35,910 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345., pid=74, masterSystemTime=1676307635875 2023-02-13 17:00:35,911 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:35,911 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:35,913 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:35,913 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:35,913 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:35,913 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f31ed543f552e93833ffd6a17e17334b columnFamilyName cf 2023-02-13 17:00:35,913 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:35,914 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7925c8f572739336270bf824c12623ea columnFamilyName cf 2023-02-13 17:00:35,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:35,915 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:35,915 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:35,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 17:00:35,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:35,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:35,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:35,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:35,917 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:35,917 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307635917"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307635917"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307635917"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307635917"}]},"ts":"1676307635917"} 2023-02-13 17:00:35,919 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=66 2023-02-13 17:00:35,919 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=66, state=SUCCESS; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 196 msec 2023-02-13 17:00:35,920 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=64 2023-02-13 17:00:35,920 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=64, state=SUCCESS; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 196 msec 2023-02-13 17:00:35,921 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN in 358 msec 2023-02-13 17:00:35,922 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN in 359 msec 2023-02-13 17:00:35,923 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=65 2023-02-13 17:00:35,923 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=65, state=SUCCESS; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670 in 197 msec 2023-02-13 17:00:35,924 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN in 362 msec 2023-02-13 17:00:35,924 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:35,926 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:35,926 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:35,927 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61523f42895e23c1110bbb6a6531b0a6 columnFamilyName cf 2023-02-13 17:00:35,932 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:35,934 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(310): Store=f31ed543f552e93833ffd6a17e17334b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:35,935 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:35,940 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(310): Store=7925c8f572739336270bf824c12623ea/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:35,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:35,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:35,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:35,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:35,942 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:35,942 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(310): Store=61523f42895e23c1110bbb6a6531b0a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:35,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:35,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:35,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:35,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:35,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f31ed543f552e93833ffd6a17e17334b; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62386358, jitterRate=-0.07037082314491272}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:35,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f31ed543f552e93833ffd6a17e17334b: Running coprocessor pre-open hook at 1676307635909Writing region info on filesystem at 1676307635909Initializing all the Stores at 1676307635911 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307635911Cleaning up temporary data from old regions at 1676307635943 (+32 ms)Cleaning up detritus from prior splits at 1676307635943Running coprocessor post-open hooks at 1676307635947 (+4 ms)Region opened successfully at 1676307635947 2023-02-13 17:00:35,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7925c8f572739336270bf824c12623ea; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68039128, jitterRate=0.013862013816833496}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:35,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7925c8f572739336270bf824c12623ea: Running coprocessor pre-open hook at 1676307635907Writing region info on filesystem at 1676307635907Initializing all the Stores at 1676307635911 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307635911Cleaning up temporary data from old regions at 1676307635943 (+32 ms)Cleaning up detritus from prior splits at 1676307635943Running coprocessor post-open hooks at 1676307635948 (+5 ms)Region opened successfully at 1676307635948 2023-02-13 17:00:35,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b., pid=75, masterSystemTime=1676307635872 2023-02-13 17:00:35,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea., pid=71, masterSystemTime=1676307635870 2023-02-13 17:00:35,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:35,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 61523f42895e23c1110bbb6a6531b0a6; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71808714, jitterRate=0.07003322243690491}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:35,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 61523f42895e23c1110bbb6a6531b0a6: Running coprocessor pre-open hook at 1676307635916Writing region info on filesystem at 1676307635916Initializing all the Stores at 1676307635918 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307635918Cleaning up temporary data from old regions at 1676307635946 (+28 ms)Cleaning up detritus from prior splits at 1676307635946Running coprocessor post-open hooks at 1676307635951 (+5 ms)Region opened successfully at 1676307635951 2023-02-13 17:00:35,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:35,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:35,952 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:35,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6., pid=73, masterSystemTime=1676307635875 2023-02-13 17:00:35,952 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307635951"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307635951"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307635951"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307635951"}]},"ts":"1676307635951"} 2023-02-13 17:00:35,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:35,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:35,953 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:35,954 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307635953"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307635953"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307635953"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307635953"}]},"ts":"1676307635953"} 2023-02-13 17:00:35,954 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:35,954 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:35,955 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:35,955 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307635955"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307635955"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307635955"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307635955"}]},"ts":"1676307635955"} 2023-02-13 17:00:35,958 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=67 2023-02-13 17:00:35,958 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=67, state=SUCCESS; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747 in 229 msec 2023-02-13 17:00:35,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=68 2023-02-13 17:00:35,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=68, state=SUCCESS; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716 in 237 msec 2023-02-13 17:00:35,960 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN in 397 msec 2023-02-13 17:00:35,960 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=69 2023-02-13 17:00:35,960 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=69, state=SUCCESS; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670 in 237 msec 2023-02-13 17:00:35,961 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN in 399 msec 2023-02-13 17:00:35,962 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=69, resume processing ppid=63 2023-02-13 17:00:35,962 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN in 399 msec 2023-02-13 17:00:35,963 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307635963"}]},"ts":"1676307635963"} 2023-02-13 17:00:35,964 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLED in hbase:meta 2023-02-13 17:00:35,967 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676307630447' was successfully enabled. 2023-02-13 17:00:35,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676307630447 in 417 msec 2023-02-13 17:00:36,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 17:00:36,306 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 63 completed 2023-02-13 17:00:36,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:36,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:36,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:36,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:36,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:36,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:36,333 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676307630447 2023-02-13 17:00:36,333 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:36,335 INFO [Listener at localhost/40167] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:00:36,586 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 17:00:36,673 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 17:00:36,673 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-13 17:00:36,770 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 17:00:36,882 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 17:00:36,986 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 17:00:37,141 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 17:00:38,587 DEBUG [Listener at localhost/40167] zookeeper.ReadOnlyZKClient(139): Connect 0x20ceba7a to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:38,593 DEBUG [Listener at localhost/40167] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14908088, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:38,596 DEBUG [hconnection-0x40c691bd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:38,598 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34746, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:38,714 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 17:00:38,718 DEBUG [hconnection-0x40c691bd-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:38,721 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42502, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:38,849 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 17:00:38,853 DEBUG [hconnection-0x40c691bd-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:38,857 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39004, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:38,947 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 17:00:39,041 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 17:00:39,167 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 17:00:40,367 INFO [Listener at localhost/40167] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:40,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676307630447 2023-02-13 17:00:40,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:40,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 17:00:40,373 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307640373"}]},"ts":"1676307640373"} 2023-02-13 17:00:40,375 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLING in hbase:meta 2023-02-13 17:00:40,377 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676307630447 to state=DISABLING 2023-02-13 17:00:40,377 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN}] 2023-02-13 17:00:40,379 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN 2023-02-13 17:00:40,379 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN 2023-02-13 17:00:40,379 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN 2023-02-13 17:00:40,380 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN 2023-02-13 17:00:40,380 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN 2023-02-13 17:00:40,380 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:40,380 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307640380"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307640380"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307640380"}]},"ts":"1676307640380"} 2023-02-13 17:00:40,380 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:40,381 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:40,380 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:40,381 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307640380"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307640380"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307640380"}]},"ts":"1676307640380"} 2023-02-13 17:00:40,381 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307640380"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307640380"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307640380"}]},"ts":"1676307640380"} 2023-02-13 17:00:40,381 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:40,381 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307640381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307640381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307640381"}]},"ts":"1676307640381"} 2023-02-13 17:00:40,381 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307640381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307640381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307640381"}]},"ts":"1676307640381"} 2023-02-13 17:00:40,382 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=82, state=RUNNABLE; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:40,383 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=81, state=RUNNABLE; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:40,384 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=77, state=RUNNABLE; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:40,385 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=80, state=RUNNABLE; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:40,386 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN 2023-02-13 17:00:40,386 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=78, state=RUNNABLE; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:40,387 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:40,387 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307640387"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307640387"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307640387"}]},"ts":"1676307640387"} 2023-02-13 17:00:40,389 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=79, state=RUNNABLE; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:40,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:40,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 61523f42895e23c1110bbb6a6531b0a6, disabling compactions & flushes 2023-02-13 17:00:40,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:40,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:40,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. after waiting 0 ms 2023-02-13 17:00:40,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:40,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 61523f42895e23c1110bbb6a6531b0a6 1/1 column families, dataSize=21.98 KB heapSize=47.50 KB 2023-02-13 17:00:40,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:40,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:40,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bb0e3d66ca9f4f9ce3e87dc8bfc74c30, disabling compactions & flushes 2023-02-13 17:00:40,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:40,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f31ed543f552e93833ffd6a17e17334b, disabling compactions & flushes 2023-02-13 17:00:40,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:40,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:40,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. after waiting 0 ms 2023-02-13 17:00:40,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:40,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:40,538 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing bb0e3d66ca9f4f9ce3e87dc8bfc74c30 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-13 17:00:40,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. after waiting 0 ms 2023-02-13 17:00:40,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:40,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f31ed543f552e93833ffd6a17e17334b 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-13 17:00:40,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 2023-02-13 17:00:40,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 2023-02-13 17:00:40,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 2023-02-13 17:00:40,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/.tmp/cf/5f02a25012d1463cb5f37ea13b2cf2dd 2023-02-13 17:00:40,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/.tmp/cf/7909cb0a0160488495640804774da893 2023-02-13 17:00:40,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/.tmp/cf/5f02a25012d1463cb5f37ea13b2cf2dd as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/5f02a25012d1463cb5f37ea13b2cf2dd 2023-02-13 17:00:40,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/.tmp/cf/f6bae3d5c579413fb7516ba3d60e58f8 2023-02-13 17:00:40,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/.tmp/cf/7909cb0a0160488495640804774da893 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/7909cb0a0160488495640804774da893 2023-02-13 17:00:40,590 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/5f02a25012d1463cb5f37ea13b2cf2dd, entries=31, sequenceid=15, filesize=10.2 K 2023-02-13 17:00:40,591 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/.tmp/cf/f6bae3d5c579413fb7516ba3d60e58f8 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/f6bae3d5c579413fb7516ba3d60e58f8 2023-02-13 17:00:40,591 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 in 53ms, sequenceid=15, compaction requested=false 2023-02-13 17:00:40,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/7909cb0a0160488495640804774da893, entries=28, sequenceid=15, filesize=9.7 K 2023-02-13 17:00:40,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for f31ed543f552e93833ffd6a17e17334b in 58ms, sequenceid=15, compaction requested=false 2023-02-13 17:00:40,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/f6bae3d5c579413fb7516ba3d60e58f8, entries=336, sequenceid=15, filesize=62.2 K 2023-02-13 17:00:40,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:00:40,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for 61523f42895e23c1110bbb6a6531b0a6 in 66ms, sequenceid=15, compaction requested=false 2023-02-13 17:00:40,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:40,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Waiting for close lock at 1676307640536Running coprocessor pre-close hooks at 1676307640536Disabling compacts and flushes for region at 1676307640536Disabling writes for close at 1676307640537 (+1 ms)Obtaining lock to block concurrent updates at 1676307640538 (+1 ms)Preparing flush snapshotting stores in bb0e3d66ca9f4f9ce3e87dc8bfc74c30 at 1676307640538Finished memstore snapshotting testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676307640539 (+1 ms)Flushing stores of testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. at 1676307640539Flushing bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: creating writer at 1676307640540 (+1 ms)Flushing mob file bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: appending metadata at 1676307640545 (+5 ms)Flushing mob file bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: closing flushed file at 1676307640545Flushing bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: appending metadata at 1676307640564 (+19 ms)Flushing bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: closing flushed file at 1676307640564Flushing bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf: reopening flushed file at 1676307640587 (+23 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 in 53ms, sequenceid=15, compaction requested=false at 1676307640591 (+4 ms)Writing region close event to WAL at 1676307640595 (+4 ms)Running coprocessor post-close hooks at 1676307640602 (+7 ms)Closed at 1676307640603 (+1 ms) 2023-02-13 17:00:40,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:40,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:40,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7925c8f572739336270bf824c12623ea, disabling compactions & flushes 2023-02-13 17:00:40,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:40,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:40,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. after waiting 0 ms 2023-02-13 17:00:40,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:40,609 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSED 2023-02-13 17:00:40,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7925c8f572739336270bf824c12623ea 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-13 17:00:40,609 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307640609"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307640609"}]},"ts":"1676307640609"} 2023-02-13 17:00:40,610 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:00:40,611 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:00:40,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:40,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f31ed543f552e93833ffd6a17e17334b: Waiting for close lock at 1676307640537Running coprocessor pre-close hooks at 1676307640537Disabling compacts and flushes for region at 1676307640537Disabling writes for close at 1676307640538 (+1 ms)Obtaining lock to block concurrent updates at 1676307640539 (+1 ms)Preparing flush snapshotting stores in f31ed543f552e93833ffd6a17e17334b at 1676307640539Finished memstore snapshotting testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676307640539Flushing stores of testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. at 1676307640540 (+1 ms)Flushing f31ed543f552e93833ffd6a17e17334b/cf: creating writer at 1676307640541 (+1 ms)Flushing mob file f31ed543f552e93833ffd6a17e17334b/cf: appending metadata at 1676307640550 (+9 ms)Flushing mob file f31ed543f552e93833ffd6a17e17334b/cf: closing flushed file at 1676307640550Flushing f31ed543f552e93833ffd6a17e17334b/cf: appending metadata at 1676307640568 (+18 ms)Flushing f31ed543f552e93833ffd6a17e17334b/cf: closing flushed file at 1676307640568Flushing f31ed543f552e93833ffd6a17e17334b/cf: reopening flushed file at 1676307640589 (+21 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for f31ed543f552e93833ffd6a17e17334b in 58ms, sequenceid=15, compaction requested=false at 1676307640596 (+7 ms)Writing region close event to WAL at 1676307640604 (+8 ms)Running coprocessor post-close hooks at 1676307640612 (+8 ms)Closed at 1676307640612 2023-02-13 17:00:40,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:40,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 61523f42895e23c1110bbb6a6531b0a6: Waiting for close lock at 1676307640535Running coprocessor pre-close hooks at 1676307640535Disabling compacts and flushes for region at 1676307640535Disabling writes for close at 1676307640536 (+1 ms)Obtaining lock to block concurrent updates at 1676307640536Preparing flush snapshotting stores in 61523f42895e23c1110bbb6a6531b0a6 at 1676307640536Finished memstore snapshotting testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6., syncing WAL and waiting on mvcc, flushsize=dataSize=22508, getHeapSize=48624, getOffHeapSize=0, getCellsCount=336 at 1676307640537 (+1 ms)Flushing stores of testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. at 1676307640538 (+1 ms)Flushing 61523f42895e23c1110bbb6a6531b0a6/cf: creating writer at 1676307640539 (+1 ms)Flushing mob file 61523f42895e23c1110bbb6a6531b0a6/cf: appending metadata at 1676307640553 (+14 ms)Flushing mob file 61523f42895e23c1110bbb6a6531b0a6/cf: closing flushed file at 1676307640553Flushing 61523f42895e23c1110bbb6a6531b0a6/cf: appending metadata at 1676307640571 (+18 ms)Flushing 61523f42895e23c1110bbb6a6531b0a6/cf: closing flushed file at 1676307640571Flushing 61523f42895e23c1110bbb6a6531b0a6/cf: reopening flushed file at 1676307640592 (+21 ms)Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for 61523f42895e23c1110bbb6a6531b0a6 in 66ms, sequenceid=15, compaction requested=false at 1676307640602 (+10 ms)Writing region close event to WAL at 1676307640606 (+4 ms)Running coprocessor post-close hooks at 1676307640612 (+6 ms)Closed at 1676307640612 2023-02-13 17:00:40,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:40,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:40,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e16e4423db3157675739f103f8f5a30, disabling compactions & flushes 2023-02-13 17:00:40,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:40,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:40,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. after waiting 0 ms 2023-02-13 17:00:40,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:40,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 0e16e4423db3157675739f103f8f5a30 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-13 17:00:40,616 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSED 2023-02-13 17:00:40,616 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=79 2023-02-13 17:00:40,616 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307640616"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307640616"}]},"ts":"1676307640616"} 2023-02-13 17:00:40,616 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=79, state=SUCCESS; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 223 msec 2023-02-13 17:00:40,617 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSED 2023-02-13 17:00:40,617 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307640617"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307640617"}]},"ts":"1676307640617"} 2023-02-13 17:00:40,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:40,618 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN in 239 msec 2023-02-13 17:00:40,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:40,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2457c5d248f99f39cff76beaa71b4345, disabling compactions & flushes 2023-02-13 17:00:40,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:40,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:40,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. after waiting 0 ms 2023-02-13 17:00:40,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:40,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 2457c5d248f99f39cff76beaa71b4345 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-13 17:00:40,622 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=80 2023-02-13 17:00:40,622 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=80, state=SUCCESS; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747 in 233 msec 2023-02-13 17:00:40,622 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=82 2023-02-13 17:00:40,622 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=82, state=SUCCESS; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670 in 238 msec 2023-02-13 17:00:40,623 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN in 245 msec 2023-02-13 17:00:40,624 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN in 245 msec 2023-02-13 17:00:40,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 17:00:40,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a 2023-02-13 17:00:40,648 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 2023-02-13 17:00:40,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 2023-02-13 17:00:40,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/.tmp/cf/d4a69b191ab64507915a551f7d3cf0ec 2023-02-13 17:00:40,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/.tmp/cf/f6d566fa52ad45b6ba62c7872adbae6b 2023-02-13 17:00:40,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/.tmp/cf/d4a69b191ab64507915a551f7d3cf0ec as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/d4a69b191ab64507915a551f7d3cf0ec 2023-02-13 17:00:40,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/.tmp/cf/46cd66f26a1c432184d8807fe83439c0 2023-02-13 17:00:40,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/.tmp/cf/f6d566fa52ad45b6ba62c7872adbae6b as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/f6d566fa52ad45b6ba62c7872adbae6b 2023-02-13 17:00:40,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/d4a69b191ab64507915a551f7d3cf0ec, entries=33, sequenceid=15, filesize=10.5 K 2023-02-13 17:00:40,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 7925c8f572739336270bf824c12623ea in 54ms, sequenceid=15, compaction requested=false 2023-02-13 17:00:40,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/.tmp/cf/46cd66f26a1c432184d8807fe83439c0 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/46cd66f26a1c432184d8807fe83439c0 2023-02-13 17:00:40,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/f6d566fa52ad45b6ba62c7872adbae6b, entries=36, sequenceid=15, filesize=11.0 K 2023-02-13 17:00:40,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/46cd66f26a1c432184d8807fe83439c0, entries=36, sequenceid=15, filesize=11.0 K 2023-02-13 17:00:40,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 2457c5d248f99f39cff76beaa71b4345 in 52ms, sequenceid=15, compaction requested=false 2023-02-13 17:00:40,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 0e16e4423db3157675739f103f8f5a30 in 58ms, sequenceid=15, compaction requested=false 2023-02-13 17:00:40,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:00:40,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:40,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7925c8f572739336270bf824c12623ea: Waiting for close lock at 1676307640608Running coprocessor pre-close hooks at 1676307640608Disabling compacts and flushes for region at 1676307640608Disabling writes for close at 1676307640609 (+1 ms)Obtaining lock to block concurrent updates at 1676307640609Preparing flush snapshotting stores in 7925c8f572739336270bf824c12623ea at 1676307640609Finished memstore snapshotting testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676307640609Flushing stores of testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. at 1676307640610 (+1 ms)Flushing 7925c8f572739336270bf824c12623ea/cf: creating writer at 1676307640610Flushing mob file 7925c8f572739336270bf824c12623ea/cf: appending metadata at 1676307640620 (+10 ms)Flushing mob file 7925c8f572739336270bf824c12623ea/cf: closing flushed file at 1676307640620Flushing 7925c8f572739336270bf824c12623ea/cf: appending metadata at 1676307640646 (+26 ms)Flushing 7925c8f572739336270bf824c12623ea/cf: closing flushed file at 1676307640646Flushing 7925c8f572739336270bf824c12623ea/cf: reopening flushed file at 1676307640659 (+13 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 7925c8f572739336270bf824c12623ea in 54ms, sequenceid=15, compaction requested=false at 1676307640664 (+5 ms)Writing region close event to WAL at 1676307640667 (+3 ms)Running coprocessor post-close hooks at 1676307640681 (+14 ms)Closed at 1676307640681 2023-02-13 17:00:40,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:00:40,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:40,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e16e4423db3157675739f103f8f5a30: Waiting for close lock at 1676307640614Running coprocessor pre-close hooks at 1676307640614Disabling compacts and flushes for region at 1676307640614Disabling writes for close at 1676307640614Obtaining lock to block concurrent updates at 1676307640614Preparing flush snapshotting stores in 0e16e4423db3157675739f103f8f5a30 at 1676307640614Finished memstore snapshotting testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676307640614Flushing stores of testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. at 1676307640615 (+1 ms)Flushing 0e16e4423db3157675739f103f8f5a30/cf: creating writer at 1676307640615Flushing mob file 0e16e4423db3157675739f103f8f5a30/cf: appending metadata at 1676307640630 (+15 ms)Flushing mob file 0e16e4423db3157675739f103f8f5a30/cf: closing flushed file at 1676307640630Flushing 0e16e4423db3157675739f103f8f5a30/cf: appending metadata at 1676307640650 (+20 ms)Flushing 0e16e4423db3157675739f103f8f5a30/cf: closing flushed file at 1676307640650Flushing 0e16e4423db3157675739f103f8f5a30/cf: reopening flushed file at 1676307640663 (+13 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 0e16e4423db3157675739f103f8f5a30 in 58ms, sequenceid=15, compaction requested=false at 1676307640672 (+9 ms)Writing region close event to WAL at 1676307640679 (+7 ms)Running coprocessor post-close hooks at 1676307640683 (+4 ms)Closed at 1676307640683 2023-02-13 17:00:40,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:40,685 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSED 2023-02-13 17:00:40,685 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307640685"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307640685"}]},"ts":"1676307640685"} 2023-02-13 17:00:40,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:40,686 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSED 2023-02-13 17:00:40,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:00:40,686 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307640686"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307640686"}]},"ts":"1676307640686"} 2023-02-13 17:00:40,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:40,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2457c5d248f99f39cff76beaa71b4345: Waiting for close lock at 1676307640618Running coprocessor pre-close hooks at 1676307640618Disabling compacts and flushes for region at 1676307640618Disabling writes for close at 1676307640618Obtaining lock to block concurrent updates at 1676307640619 (+1 ms)Preparing flush snapshotting stores in 2457c5d248f99f39cff76beaa71b4345 at 1676307640619Finished memstore snapshotting testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676307640619Flushing stores of testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. at 1676307640619Flushing 2457c5d248f99f39cff76beaa71b4345/cf: creating writer at 1676307640628 (+9 ms)Flushing mob file 2457c5d248f99f39cff76beaa71b4345/cf: appending metadata at 1676307640632 (+4 ms)Flushing mob file 2457c5d248f99f39cff76beaa71b4345/cf: closing flushed file at 1676307640632Flushing 2457c5d248f99f39cff76beaa71b4345/cf: appending metadata at 1676307640649 (+17 ms)Flushing 2457c5d248f99f39cff76beaa71b4345/cf: closing flushed file at 1676307640649Flushing 2457c5d248f99f39cff76beaa71b4345/cf: reopening flushed file at 1676307640667 (+18 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 2457c5d248f99f39cff76beaa71b4345 in 52ms, sequenceid=15, compaction requested=false at 1676307640671 (+4 ms)Writing region close event to WAL at 1676307640679 (+8 ms)Running coprocessor post-close hooks at 1676307640687 (+8 ms)Closed at 1676307640687 2023-02-13 17:00:40,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:40,690 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSED 2023-02-13 17:00:40,690 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307640689"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307640689"}]},"ts":"1676307640689"} 2023-02-13 17:00:40,690 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=81 2023-02-13 17:00:40,691 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=81, state=SUCCESS; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716 in 304 msec 2023-02-13 17:00:40,691 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=77 2023-02-13 17:00:40,691 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=77, state=SUCCESS; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 304 msec 2023-02-13 17:00:40,692 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN in 314 msec 2023-02-13 17:00:40,693 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN in 314 msec 2023-02-13 17:00:40,693 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=78 2023-02-13 17:00:40,693 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=78, state=SUCCESS; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670 in 306 msec 2023-02-13 17:00:40,695 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=78, resume processing ppid=76 2023-02-13 17:00:40,695 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN in 316 msec 2023-02-13 17:00:40,696 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307640696"}]},"ts":"1676307640696"} 2023-02-13 17:00:40,697 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLED in hbase:meta 2023-02-13 17:00:40,700 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676307630447 to state=DISABLED 2023-02-13 17:00:40,702 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676307630447 in 333 msec 2023-02-13 17:00:41,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 17:00:41,125 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 76 completed 2023-02-13 17:00:41,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:41,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676307641126 (current time:1676307641126). 2023-02-13 17:00:41,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 17:00:41,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(139): Connect 0x7b09c896 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:41,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a213e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:41,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:41,136 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34750, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:41,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7b09c896 to 127.0.0.1:54083 2023-02-13 17:00:41,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:41,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 17:00:41,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:41,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:41,141 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:41,141 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE; timeout=600000, timestamp=1676308241141 2023-02-13 17:00:41,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:41,141 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676307630447 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 2023-02-13 17:00:41,142 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 17:00:41,142 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 17:00:41,163 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } 2023-02-13 17:00:41,169 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,169 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,169 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,169 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,169 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,169 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,170 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,170 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,170 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,170 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,173 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/46cd66f26a1c432184d8807fe83439c0] hfiles 2023-02-13 17:00:41,173 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:41,173 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/46cd66f26a1c432184d8807fe83439c0 2023-02-13 17:00:41,174 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/d4a69b191ab64507915a551f7d3cf0ec] hfiles 2023-02-13 17:00:41,174 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:41,174 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/d4a69b191ab64507915a551f7d3cf0ec 2023-02-13 17:00:41,170 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,173 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/f6d566fa52ad45b6ba62c7872adbae6b] hfiles 2023-02-13 17:00:41,175 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/5f02a25012d1463cb5f37ea13b2cf2dd, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9] hfiles 2023-02-13 17:00:41,175 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,175 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/5f02a25012d1463cb5f37ea13b2cf2dd 2023-02-13 17:00:41,175 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:41,176 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/f6d566fa52ad45b6ba62c7872adbae6b 2023-02-13 17:00:41,176 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:41,176 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/7909cb0a0160488495640804774da893] hfiles 2023-02-13 17:00:41,176 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:41,176 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/7909cb0a0160488495640804774da893 2023-02-13 17:00:41,177 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/f6bae3d5c579413fb7516ba3d60e58f8] hfiles 2023-02-13 17:00:41,177 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:41,177 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/f6bae3d5c579413fb7516ba3d60e58f8 2023-02-13 17:00:41,196 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1] hfiles 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 2023-02-13 17:00:41,197 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 2023-02-13 17:00:41,250 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } as finished. 2023-02-13 17:00:41,250 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676307630447 2023-02-13 17:00:41,251 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb1-1676307630447 2023-02-13 17:00:41,282 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb1-1676307630447 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/snaptb1-1676307630447 2023-02-13 17:00:41,301 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676307630447 of table testRestoreSnapshot-1676307630447 completed 2023-02-13 17:00:41,301 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb1-1676307630447 2023-02-13 17:00:41,302 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb1-1676307630447 2023-02-13 17:00:41,302 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676307630447 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 at 1676307641142Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } at 1676307641163 (+21 ms)Consolidate snapshot: snaptb1-1676307630447 at 1676307641250 (+87 ms)Loading Region manifests for snaptb1-1676307630447 at 1676307641250Writing data manifest for snaptb1-1676307630447 at 1676307641255 (+5 ms)Verifying snapshot: snaptb1-1676307630447 at 1676307641266 (+11 ms)Snapshot snaptb1-1676307630447 of table testRestoreSnapshot-1676307630447 completed at 1676307641301 (+35 ms) 2023-02-13 17:00:41,303 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:41,304 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE in 164 msec 2023-02-13 17:00:41,378 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 17:00:41,392 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 17:00:41,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } is done 2023-02-13 17:00:41,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 17:00:41,412 INFO [Listener at localhost/40167] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676307630447-1676307641412 2023-02-13 17:00:41,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676307630447-1676307641412 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:41,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676307641413 (current time:1676307641413). 2023-02-13 17:00:41,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 17:00:41,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(139): Connect 0x18c4517f to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:41,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1cbbcf9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:41,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:41,424 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34754, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:41,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18c4517f to 127.0.0.1:54083 2023-02-13 17:00:41,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:41,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 17:00:41,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676307630447-1676307641412 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:41,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:41,428 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:41,429 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE; timeout=600000, timestamp=1676308241429 2023-02-13 17:00:41,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676307630447-1676307641412 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:41,429 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676307630447-1676307641412 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 2023-02-13 17:00:41,429 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676307630447-1676307641412 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 17:00:41,429 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 17:00:41,441 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676307630447-1676307641412 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } 2023-02-13 17:00:41,445 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,445 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,445 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,445 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,445 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,445 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,445 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,446 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,446 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,446 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,446 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,446 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,448 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/d4a69b191ab64507915a551f7d3cf0ec] hfiles 2023-02-13 17:00:41,448 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:41,448 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/d4a69b191ab64507915a551f7d3cf0ec 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/f6bae3d5c579413fb7516ba3d60e58f8] hfiles 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/f6bae3d5c579413fb7516ba3d60e58f8 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/46cd66f26a1c432184d8807fe83439c0] hfiles 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/46cd66f26a1c432184d8807fe83439c0 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/7909cb0a0160488495640804774da893] hfiles 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/7909cb0a0160488495640804774da893 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/5f02a25012d1463cb5f37ea13b2cf2dd, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9] hfiles 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/5f02a25012d1463cb5f37ea13b2cf2dd 2023-02-13 17:00:41,449 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:41,450 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:41,450 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:41,458 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/f6d566fa52ad45b6ba62c7872adbae6b] hfiles 2023-02-13 17:00:41,458 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:41,458 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/f6d566fa52ad45b6ba62c7872adbae6b 2023-02-13 17:00:41,459 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1] hfiles 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 2023-02-13 17:00:41,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 2023-02-13 17:00:41,489 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676307630447-1676307641412 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } as finished. 2023-02-13 17:00:41,489 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676307630447-1676307641412 2023-02-13 17:00:41,490 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676307630447-1676307641412 2023-02-13 17:00:41,518 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ceba7a to 127.0.0.1:54083 2023-02-13 17:00:41,518 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:41,548 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676307630447-1676307641412 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/hbase-failsafe-snaptb0-1676307630447-1676307641412 2023-02-13 17:00:41,569 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676307630447-1676307641412 of table testRestoreSnapshot-1676307630447 completed 2023-02-13 17:00:41,569 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676307630447-1676307641412 2023-02-13 17:00:41,569 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676307630447-1676307641412 2023-02-13 17:00:41,569 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676307630447-1676307641412 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 at 1676307641429Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676307630447-1676307641412 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } at 1676307641441 (+12 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676307630447-1676307641412 at 1676307641489 (+48 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676307630447-1676307641412 at 1676307641489Writing data manifest for hbase-failsafe-snaptb0-1676307630447-1676307641412 at 1676307641520 (+31 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676307630447-1676307641412 at 1676307641536 (+16 ms)Snapshot hbase-failsafe-snaptb0-1676307630447-1676307641412 of table testRestoreSnapshot-1676307630447 completed at 1676307641569 (+33 ms) 2023-02-13 17:00:41,571 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:41,572 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE in 145 msec 2023-02-13 17:00:41,679 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 17:00:41,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676307630447-1676307641412 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } is done 2023-02-13 17:00:41,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676307630447-1676307641412 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 17:00:41,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676307630447 snapshot=name: "snaptb0-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307634777 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 17:00:41,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676307630447 as table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:41,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-13 17:00:41,727 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000002 2023-02-13 17:00:41,728 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000001 2023-02-13 17:00:41,728 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000002 2023-02-13 17:00:41,729 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } 2023-02-13 17:00:41,733 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307634777 type: DISABLED version: 2 ttl: 0 2023-02-13 17:00:41,734 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447 2023-02-13 17:00:41,743 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:41,750 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 2023-02-13 17:00:41,752 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a 2023-02-13 17:00:41,754 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 2023-02-13 17:00:41,755 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 2023-02-13 17:00:41,756 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 2023-02-13 17:00:41,757 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 2023-02-13 17:00:41,757 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:41,757 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:41,757 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:41,757 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:41,758 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:41,758 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:41,763 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/f6bae3d5c579413fb7516ba3d60e58f8 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/f6bae3d5c579413fb7516ba3d60e58f8 2023-02-13 17:00:41,763 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/d4a69b191ab64507915a551f7d3cf0ec to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/d4a69b191ab64507915a551f7d3cf0ec 2023-02-13 17:00:41,763 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/f6d566fa52ad45b6ba62c7872adbae6b to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/f6d566fa52ad45b6ba62c7872adbae6b 2023-02-13 17:00:41,764 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/5f02a25012d1463cb5f37ea13b2cf2dd to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/5f02a25012d1463cb5f37ea13b2cf2dd 2023-02-13 17:00:41,764 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/46cd66f26a1c432184d8807fe83439c0 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/46cd66f26a1c432184d8807fe83439c0 2023-02-13 17:00:41,764 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/7909cb0a0160488495640804774da893 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/7909cb0a0160488495640804774da893 2023-02-13 17:00:41,765 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307634777 type: DISABLED version: 2 ttl: 0 2023-02-13 17:00:41,766 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307641766"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:41,766 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307641766"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:41,766 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307641766"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:41,767 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307641766"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:41,767 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307641766"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:41,767 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307641766"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:41,770 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 17:00:41,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 17:00:41,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307641767"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307641767"}]},"ts":"1676307641767"} 2023-02-13 17:00:41,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307641767"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307641767"}]},"ts":"1676307641767"} 2023-02-13 17:00:41,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307641767"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307641767"}]},"ts":"1676307641767"} 2023-02-13 17:00:41,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307641767"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307641767"}]},"ts":"1676307641767"} 2023-02-13 17:00:41,771 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307641767"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307641767"}]},"ts":"1676307641767"} 2023-02-13 17:00:41,771 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307641767"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307641767"}]},"ts":"1676307641767"} 2023-02-13 17:00:41,773 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 17:00:41,773 INFO [PEWorker-2] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 17:00:41,773 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 17:00:41,774 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676307630447 completed! 2023-02-13 17:00:41,776 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676307630447 snapshot=name: "snaptb0-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307634777 type: DISABLED version: 2 ttl: 0 ) in 77 msec 2023-02-13 17:00:41,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-13 17:00:41,962 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676307630447, procId: 91 completed 2023-02-13 17:00:41,962 INFO [Listener at localhost/40167] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676307630447-1676307641412 2023-02-13 17:00:41,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1676307630447-1676307641412" 2023-02-13 17:00:41,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676307630447-1676307641412 2023-02-13 17:00:41,970 INFO [Listener at localhost/40167] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:41,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676307630447 2023-02-13 17:00:41,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:41,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 17:00:41,975 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676307630447 2023-02-13 17:00:41,975 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307641975"}]},"ts":"1676307641975"} 2023-02-13 17:00:41,976 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLING in hbase:meta 2023-02-13 17:00:41,979 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 17:00:41,981 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN}] 2023-02-13 17:00:41,983 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN 2023-02-13 17:00:41,983 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN 2023-02-13 17:00:41,983 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN 2023-02-13 17:00:41,983 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN 2023-02-13 17:00:41,984 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:41,985 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:41,985 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:41,985 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:41,985 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN 2023-02-13 17:00:41,986 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN 2023-02-13 17:00:41,986 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:41,987 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:42,134 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:00:42,135 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:00:42,135 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:00:42,135 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:00:42,135 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:00:42,138 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:42,138 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:42,138 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:42,138 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:42,138 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307642138"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307642138"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307642138"}]},"ts":"1676307642138"} 2023-02-13 17:00:42,138 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307642138"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307642138"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307642138"}]},"ts":"1676307642138"} 2023-02-13 17:00:42,138 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:42,138 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307642138"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307642138"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307642138"}]},"ts":"1676307642138"} 2023-02-13 17:00:42,138 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307642138"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307642138"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307642138"}]},"ts":"1676307642138"} 2023-02-13 17:00:42,138 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307642138"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307642138"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307642138"}]},"ts":"1676307642138"} 2023-02-13 17:00:42,140 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=94, state=RUNNABLE; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:42,141 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=96, state=RUNNABLE; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:42,142 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=97, state=RUNNABLE; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:42,143 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=95, state=RUNNABLE; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:42,143 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=98, state=RUNNABLE; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:42,143 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:42,144 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307642143"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307642143"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307642143"}]},"ts":"1676307642143"} 2023-02-13 17:00:42,146 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=93, state=RUNNABLE; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:42,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 17:00:42,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:42,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 17:00:42,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:42,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:42,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:42,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 17:00:42,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:42,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:42,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:42,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:42,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:42,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:42,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:42,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 17:00:42,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:42,297 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:42,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:42,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:42,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:42,297 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:42,298 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:42,298 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:42,298 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:42,298 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:42,298 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2457c5d248f99f39cff76beaa71b4345 columnFamilyName cf 2023-02-13 17:00:42,298 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f31ed543f552e93833ffd6a17e17334b columnFamilyName cf 2023-02-13 17:00:42,298 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:42,300 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:42,300 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:42,300 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61523f42895e23c1110bbb6a6531b0a6 columnFamilyName cf 2023-02-13 17:00:42,304 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:42,304 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(310): Store=2457c5d248f99f39cff76beaa71b4345/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:42,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:42,306 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:42,306 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(310): Store=61523f42895e23c1110bbb6a6531b0a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:42,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:42,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:42,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:42,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:42,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2457c5d248f99f39cff76beaa71b4345; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74867993, jitterRate=0.11562003195285797}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:42,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2457c5d248f99f39cff76beaa71b4345: Running coprocessor pre-open hook at 1676307642296Writing region info on filesystem at 1676307642296Initializing all the Stores at 1676307642296Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307642296Cleaning up temporary data from old regions at 1676307642307 (+11 ms)Cleaning up detritus from prior splits at 1676307642308 (+1 ms)Running coprocessor post-open hooks at 1676307642311 (+3 ms)Region opened successfully at 1676307642311 2023-02-13 17:00:42,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:42,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345., pid=99, masterSystemTime=1676307642292 2023-02-13 17:00:42,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 61523f42895e23c1110bbb6a6531b0a6; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61284822, jitterRate=-0.08678498864173889}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:42,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 61523f42895e23c1110bbb6a6531b0a6: Running coprocessor pre-open hook at 1676307642297Writing region info on filesystem at 1676307642297Initializing all the Stores at 1676307642298 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307642298Cleaning up temporary data from old regions at 1676307642309 (+11 ms)Cleaning up detritus from prior splits at 1676307642309Running coprocessor post-open hooks at 1676307642312 (+3 ms)Region opened successfully at 1676307642312 2023-02-13 17:00:42,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6., pid=103, masterSystemTime=1676307642294 2023-02-13 17:00:42,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:42,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:42,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:42,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 17:00:42,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:42,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:42,314 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:42,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:42,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:42,314 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307642314"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307642314"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307642314"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307642314"}]},"ts":"1676307642314"} 2023-02-13 17:00:42,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:42,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:42,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:42,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 17:00:42,315 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:42,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:42,315 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307642315"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307642315"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307642315"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307642315"}]},"ts":"1676307642315"} 2023-02-13 17:00:42,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:42,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:42,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:42,316 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:42,316 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:42,317 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:42,318 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(310): Store=f31ed543f552e93833ffd6a17e17334b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:42,318 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:42,318 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:42,319 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7925c8f572739336270bf824c12623ea columnFamilyName cf 2023-02-13 17:00:42,319 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:42,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:42,319 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=94 2023-02-13 17:00:42,319 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=94, state=SUCCESS; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43065,1676307627716 in 176 msec 2023-02-13 17:00:42,319 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:42,319 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 columnFamilyName cf 2023-02-13 17:00:42,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:42,321 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=98 2023-02-13 17:00:42,321 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=98, state=SUCCESS; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670 in 174 msec 2023-02-13 17:00:42,321 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN in 340 msec 2023-02-13 17:00:42,322 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN in 340 msec 2023-02-13 17:00:42,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:42,325 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f31ed543f552e93833ffd6a17e17334b; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72463982, jitterRate=0.07979747653007507}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:42,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f31ed543f552e93833ffd6a17e17334b: Running coprocessor pre-open hook at 1676307642296Writing region info on filesystem at 1676307642296Initializing all the Stores at 1676307642297 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307642297Cleaning up temporary data from old regions at 1676307642321 (+24 ms)Cleaning up detritus from prior splits at 1676307642322 (+1 ms)Running coprocessor post-open hooks at 1676307642325 (+3 ms)Region opened successfully at 1676307642325 2023-02-13 17:00:42,326 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b., pid=100, masterSystemTime=1676307642293 2023-02-13 17:00:42,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:42,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:42,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:42,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 17:00:42,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:42,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:42,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:42,329 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:42,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:42,329 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(310): Store=7925c8f572739336270bf824c12623ea/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:42,329 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:42,329 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307642329"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307642329"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307642329"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307642329"}]},"ts":"1676307642329"} 2023-02-13 17:00:42,332 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:42,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:42,333 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:42,333 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:42,333 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:42,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=96 2023-02-13 17:00:42,333 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(310): Store=bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:42,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=96, state=SUCCESS; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747 in 190 msec 2023-02-13 17:00:42,334 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e16e4423db3157675739f103f8f5a30 columnFamilyName cf 2023-02-13 17:00:42,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:42,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:42,335 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN in 352 msec 2023-02-13 17:00:42,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:42,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:42,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:42,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7925c8f572739336270bf824c12623ea; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71924156, jitterRate=0.07175344228744507}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:42,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7925c8f572739336270bf824c12623ea: Running coprocessor pre-open hook at 1676307642314Writing region info on filesystem at 1676307642314Initializing all the Stores at 1676307642316 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307642316Cleaning up temporary data from old regions at 1676307642335 (+19 ms)Cleaning up detritus from prior splits at 1676307642336 (+1 ms)Running coprocessor post-open hooks at 1676307642339 (+3 ms)Region opened successfully at 1676307642339 2023-02-13 17:00:42,339 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:42,339 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(310): Store=0e16e4423db3157675739f103f8f5a30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:42,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea., pid=101, masterSystemTime=1676307642292 2023-02-13 17:00:42,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bb0e3d66ca9f4f9ce3e87dc8bfc74c30; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74219247, jitterRate=0.10595296323299408}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:42,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Running coprocessor pre-open hook at 1676307642315Writing region info on filesystem at 1676307642315Initializing all the Stores at 1676307642316 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307642316Cleaning up temporary data from old regions at 1676307642336 (+20 ms)Cleaning up detritus from prior splits at 1676307642337 (+1 ms)Running coprocessor post-open hooks at 1676307642340 (+3 ms)Region opened successfully at 1676307642340 2023-02-13 17:00:42,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:42,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30., pid=102, masterSystemTime=1676307642294 2023-02-13 17:00:42,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:42,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:42,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:42,342 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:42,342 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307642342"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307642342"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307642342"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307642342"}]},"ts":"1676307642342"} 2023-02-13 17:00:42,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:42,343 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:42,343 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:42,343 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307642343"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307642343"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307642343"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307642343"}]},"ts":"1676307642343"} 2023-02-13 17:00:42,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=97 2023-02-13 17:00:42,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=97, state=SUCCESS; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716 in 202 msec 2023-02-13 17:00:42,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:42,347 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=95 2023-02-13 17:00:42,347 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=95, state=SUCCESS; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43327,1676307627670 in 203 msec 2023-02-13 17:00:42,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e16e4423db3157675739f103f8f5a30; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60962872, jitterRate=-0.09158241748809814}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:42,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e16e4423db3157675739f103f8f5a30: Running coprocessor pre-open hook at 1676307642329Writing region info on filesystem at 1676307642329Initializing all the Stores at 1676307642330 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307642330Cleaning up temporary data from old regions at 1676307642343 (+13 ms)Cleaning up detritus from prior splits at 1676307642344 (+1 ms)Running coprocessor post-open hooks at 1676307642348 (+4 ms)Region opened successfully at 1676307642348 2023-02-13 17:00:42,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30., pid=104, masterSystemTime=1676307642293 2023-02-13 17:00:42,349 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN in 365 msec 2023-02-13 17:00:42,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN in 366 msec 2023-02-13 17:00:42,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:42,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:42,350 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:42,350 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307642350"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307642350"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307642350"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307642350"}]},"ts":"1676307642350"} 2023-02-13 17:00:42,353 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=93 2023-02-13 17:00:42,353 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=93, state=SUCCESS; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 206 msec 2023-02-13 17:00:42,355 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=93, resume processing ppid=92 2023-02-13 17:00:42,355 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN in 374 msec 2023-02-13 17:00:42,356 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307642356"}]},"ts":"1676307642356"} 2023-02-13 17:00:42,357 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLED in hbase:meta 2023-02-13 17:00:42,359 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676307630447' was successfully enabled. 2023-02-13 17:00:42,361 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676307630447 in 388 msec 2023-02-13 17:00:42,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 17:00:42,727 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 92 completed 2023-02-13 17:00:42,727 DEBUG [Listener at localhost/40167] zookeeper.ReadOnlyZKClient(139): Connect 0x3e907963 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:42,733 DEBUG [Listener at localhost/40167] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@116b3a18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:42,735 DEBUG [hconnection-0x5587b766-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:42,738 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34760, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:42,845 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 17:00:42,848 DEBUG [hconnection-0x5587b766-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:42,854 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39010, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:42,972 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 17:00:42,975 DEBUG [hconnection-0x5587b766-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:42,979 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42506, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:43,054 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 17:00:43,141 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 17:00:43,254 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 17:00:44,263 INFO [Listener at localhost/40167] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:44,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676307630447 2023-02-13 17:00:44,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:44,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 17:00:44,269 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307644269"}]},"ts":"1676307644269"} 2023-02-13 17:00:44,270 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLING in hbase:meta 2023-02-13 17:00:44,272 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676307630447 to state=DISABLING 2023-02-13 17:00:44,273 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN}] 2023-02-13 17:00:44,276 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN 2023-02-13 17:00:44,276 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN 2023-02-13 17:00:44,276 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN 2023-02-13 17:00:44,276 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN 2023-02-13 17:00:44,276 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN 2023-02-13 17:00:44,277 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:44,277 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:44,277 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:44,277 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:44,277 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:44,278 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307644277"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307644277"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307644277"}]},"ts":"1676307644277"} 2023-02-13 17:00:44,278 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644277"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307644277"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307644277"}]},"ts":"1676307644277"} 2023-02-13 17:00:44,278 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644277"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307644277"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307644277"}]},"ts":"1676307644277"} 2023-02-13 17:00:44,277 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644277"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307644277"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307644277"}]},"ts":"1676307644277"} 2023-02-13 17:00:44,277 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307644277"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307644277"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307644277"}]},"ts":"1676307644277"} 2023-02-13 17:00:44,279 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=107, state=RUNNABLE; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:44,282 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=111, state=RUNNABLE; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:44,283 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=108, state=RUNNABLE; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:44,284 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=109, state=RUNNABLE; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:44,284 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN 2023-02-13 17:00:44,285 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=106, state=RUNNABLE; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:44,285 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:44,285 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644285"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307644285"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307644285"}]},"ts":"1676307644285"} 2023-02-13 17:00:44,287 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=110, state=RUNNABLE; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:44,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:44,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7925c8f572739336270bf824c12623ea, disabling compactions & flushes 2023-02-13 17:00:44,434 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:44,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:44,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. after waiting 0 ms 2023-02-13 17:00:44,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:44,434 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:44,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bb0e3d66ca9f4f9ce3e87dc8bfc74c30, disabling compactions & flushes 2023-02-13 17:00:44,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:44,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:44,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. after waiting 0 ms 2023-02-13 17:00:44,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:44,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:44,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f31ed543f552e93833ffd6a17e17334b, disabling compactions & flushes 2023-02-13 17:00:44,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:44,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:44,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. after waiting 0 ms 2023-02-13 17:00:44,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:44,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 17:00:44,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 17:00:44,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 17:00:44,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:44,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7925c8f572739336270bf824c12623ea: Waiting for close lock at 1676307644432Running coprocessor pre-close hooks at 1676307644432Disabling compacts and flushes for region at 1676307644432Disabling writes for close at 1676307644434 (+2 ms)Writing region close event to WAL at 1676307644438 (+4 ms)Running coprocessor post-close hooks at 1676307644446 (+8 ms)Closed at 1676307644446 2023-02-13 17:00:44,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:44,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Waiting for close lock at 1676307644434Running coprocessor pre-close hooks at 1676307644434Disabling compacts and flushes for region at 1676307644434Disabling writes for close at 1676307644438 (+4 ms)Writing region close event to WAL at 1676307644443 (+5 ms)Running coprocessor post-close hooks at 1676307644447 (+4 ms)Closed at 1676307644447 2023-02-13 17:00:44,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:44,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f31ed543f552e93833ffd6a17e17334b: Waiting for close lock at 1676307644438Running coprocessor pre-close hooks at 1676307644438Disabling compacts and flushes for region at 1676307644438Disabling writes for close at 1676307644438Writing region close event to WAL at 1676307644444 (+6 ms)Running coprocessor post-close hooks at 1676307644447 (+3 ms)Closed at 1676307644447 2023-02-13 17:00:44,448 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:44,448 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:44,448 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2457c5d248f99f39cff76beaa71b4345, disabling compactions & flushes 2023-02-13 17:00:44,448 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:44,448 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:44,448 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. after waiting 0 ms 2023-02-13 17:00:44,448 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:44,448 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSED 2023-02-13 17:00:44,449 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644448"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644448"}]},"ts":"1676307644448"} 2023-02-13 17:00:44,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:44,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:44,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 61523f42895e23c1110bbb6a6531b0a6, disabling compactions & flushes 2023-02-13 17:00:44,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:44,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:44,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. after waiting 0 ms 2023-02-13 17:00:44,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:44,449 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSED 2023-02-13 17:00:44,450 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644449"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644449"}]},"ts":"1676307644449"} 2023-02-13 17:00:44,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:44,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:44,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e16e4423db3157675739f103f8f5a30, disabling compactions & flushes 2023-02-13 17:00:44,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:44,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:44,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. after waiting 0 ms 2023-02-13 17:00:44,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:44,452 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSED 2023-02-13 17:00:44,453 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644452"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644452"}]},"ts":"1676307644452"} 2023-02-13 17:00:44,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 17:00:44,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 17:00:44,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:44,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2457c5d248f99f39cff76beaa71b4345: Waiting for close lock at 1676307644448Running coprocessor pre-close hooks at 1676307644448Disabling compacts and flushes for region at 1676307644448Disabling writes for close at 1676307644448Writing region close event to WAL at 1676307644449 (+1 ms)Running coprocessor post-close hooks at 1676307644454 (+5 ms)Closed at 1676307644454 2023-02-13 17:00:44,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:44,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 61523f42895e23c1110bbb6a6531b0a6: Waiting for close lock at 1676307644449Running coprocessor pre-close hooks at 1676307644449Disabling compacts and flushes for region at 1676307644449Disabling writes for close at 1676307644449Writing region close event to WAL at 1676307644450 (+1 ms)Running coprocessor post-close hooks at 1676307644454 (+4 ms)Closed at 1676307644454 2023-02-13 17:00:44,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 17:00:44,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:44,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e16e4423db3157675739f103f8f5a30: Waiting for close lock at 1676307644450Running coprocessor pre-close hooks at 1676307644450Disabling compacts and flushes for region at 1676307644450Disabling writes for close at 1676307644450Writing region close event to WAL at 1676307644451 (+1 ms)Running coprocessor post-close hooks at 1676307644456 (+5 ms)Closed at 1676307644456 2023-02-13 17:00:44,458 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=110 2023-02-13 17:00:44,458 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=110, state=SUCCESS; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43065,1676307627716 in 163 msec 2023-02-13 17:00:44,458 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=108 2023-02-13 17:00:44,458 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=108, state=SUCCESS; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43327,1676307627670 in 168 msec 2023-02-13 17:00:44,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:44,459 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSED 2023-02-13 17:00:44,459 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644459"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644459"}]},"ts":"1676307644459"} 2023-02-13 17:00:44,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:44,459 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSED 2023-02-13 17:00:44,459 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN in 185 msec 2023-02-13 17:00:44,459 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307644459"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644459"}]},"ts":"1676307644459"} 2023-02-13 17:00:44,460 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN in 185 msec 2023-02-13 17:00:44,461 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:44,461 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=109 2023-02-13 17:00:44,461 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSED 2023-02-13 17:00:44,461 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=109, state=SUCCESS; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747 in 175 msec 2023-02-13 17:00:44,462 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307644461"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644461"}]},"ts":"1676307644461"} 2023-02-13 17:00:44,463 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=107 2023-02-13 17:00:44,463 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=107, state=SUCCESS; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43065,1676307627716 in 181 msec 2023-02-13 17:00:44,463 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=111 2023-02-13 17:00:44,463 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN in 188 msec 2023-02-13 17:00:44,463 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=111, state=SUCCESS; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670 in 181 msec 2023-02-13 17:00:44,464 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN in 190 msec 2023-02-13 17:00:44,464 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN in 190 msec 2023-02-13 17:00:44,464 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=106 2023-02-13 17:00:44,464 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=106, state=SUCCESS; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 178 msec 2023-02-13 17:00:44,466 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=106, resume processing ppid=105 2023-02-13 17:00:44,466 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN in 191 msec 2023-02-13 17:00:44,466 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307644466"}]},"ts":"1676307644466"} 2023-02-13 17:00:44,468 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLED in hbase:meta 2023-02-13 17:00:44,469 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676307630447 to state=DISABLED 2023-02-13 17:00:44,471 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676307630447 in 206 msec 2023-02-13 17:00:44,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 17:00:44,519 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 105 completed 2023-02-13 17:00:44,528 INFO [Listener at localhost/40167] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676307630447-1676307644528 2023-02-13 17:00:44,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676307630447-1676307644528 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:44,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676307644528 (current time:1676307644528). 2023-02-13 17:00:44,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 17:00:44,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(139): Connect 0x4c3b6f7e to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:44,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@507f711b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:44,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:44,538 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55244, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:44,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c3b6f7e to 127.0.0.1:54083 2023-02-13 17:00:44,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:44,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 17:00:44,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676307630447-1676307644528 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:44,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:44,542 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:44,542 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE; timeout=600000, timestamp=1676308244542 2023-02-13 17:00:44,542 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676307630447-1676307644528 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 2023-02-13 17:00:44,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676307630447-1676307644528 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:44,543 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676307630447-1676307644528 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 17:00:44,543 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 17:00:44,554 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676307630447-1676307644528 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } 2023-02-13 17:00:44,556 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:44,556 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:44,556 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:44,556 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:44,556 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:44,556 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:44,556 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:44,556 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:44,557 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:44,557 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:44,557 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:44,557 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:44,558 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481] hfiles 2023-02-13 17:00:44,558 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a] hfiles 2023-02-13 17:00:44,559 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:44,559 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a] hfiles 2023-02-13 17:00:44,559 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:44,559 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:44,559 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d] hfiles 2023-02-13 17:00:44,559 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:44,560 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d] hfiles 2023-02-13 17:00:44,560 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:44,562 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:44,562 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:44,568 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1] hfiles 2023-02-13 17:00:44,568 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 2023-02-13 17:00:44,568 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9] hfiles 2023-02-13 17:00:44,568 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 2023-02-13 17:00:44,568 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:44,568 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 2023-02-13 17:00:44,568 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 2023-02-13 17:00:44,569 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 2023-02-13 17:00:44,569 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 2023-02-13 17:00:44,599 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676307630447-1676307644528 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } as finished. 2023-02-13 17:00:44,599 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676307630447-1676307644528 2023-02-13 17:00:44,600 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676307630447-1676307644528 2023-02-13 17:00:44,620 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676307630447-1676307644528 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676307630447-1676307644528 2023-02-13 17:00:44,655 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676307630447-1676307644528 of table testRestoreSnapshot-1676307630447 completed 2023-02-13 17:00:44,655 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676307630447-1676307644528 2023-02-13 17:00:44,656 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676307630447-1676307644528 2023-02-13 17:00:44,656 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676307630447-1676307644528 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 at 1676307644542Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676307630447-1676307644528 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } at 1676307644554 (+12 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676307630447-1676307644528 at 1676307644599 (+45 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676307630447-1676307644528 at 1676307644599Writing data manifest for hbase-failsafe-emptySnaptb-1676307630447-1676307644528 at 1676307644603 (+4 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676307630447-1676307644528 at 1676307644612 (+9 ms)Snapshot hbase-failsafe-emptySnaptb-1676307630447-1676307644528 of table testRestoreSnapshot-1676307630447 completed at 1676307644655 (+43 ms) 2023-02-13 17:00:44,658 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:44,659 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE in 118 msec 2023-02-13 17:00:44,793 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 17:00:44,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676307630447-1676307644528 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } is done 2023-02-13 17:00:44,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676307630447-1676307644528 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 17:00:44,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676307630447 snapshot=name: "emptySnaptb-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307631586 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 17:00:44,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676307630447 as table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:44,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-13 17:00:44,813 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000003 2023-02-13 17:00:44,814 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000002 2023-02-13 17:00:44,814 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000003 2023-02-13 17:00:44,815 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } 2023-02-13 17:00:44,816 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307631586 type: DISABLED version: 2 ttl: 0 2023-02-13 17:00:44,816 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447 2023-02-13 17:00:44,823 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:44,826 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 2023-02-13 17:00:44,827 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 2023-02-13 17:00:44,828 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 2023-02-13 17:00:44,830 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 2023-02-13 17:00:44,831 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 2023-02-13 17:00:44,832 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 2023-02-13 17:00:44,832 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:44,832 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:44,832 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:44,832 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:44,832 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:44,832 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:44,838 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:44,838 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:44,838 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:44,838 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:44,839 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:44,839 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:44,840 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307631586 type: DISABLED version: 2 ttl: 0 2023-02-13 17:00:44,841 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307644841"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:44,841 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307644841"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:44,841 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307644841"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:44,841 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307644841"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:44,841 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307644841"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:44,841 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307644841"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:44,844 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 17:00:44,844 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 17:00:44,844 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307644842"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644842"}]},"ts":"1676307644842"} 2023-02-13 17:00:44,844 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644842"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644842"}]},"ts":"1676307644842"} 2023-02-13 17:00:44,844 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307644842"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644842"}]},"ts":"1676307644842"} 2023-02-13 17:00:44,844 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644842"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644842"}]},"ts":"1676307644842"} 2023-02-13 17:00:44,845 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644842"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644842"}]},"ts":"1676307644842"} 2023-02-13 17:00:44,845 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307644842"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307644842"}]},"ts":"1676307644842"} 2023-02-13 17:00:44,847 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 17:00:44,847 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 17:00:44,847 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 17:00:44,847 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676307630447 completed! 2023-02-13 17:00:44,849 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676307630447 snapshot=name: "emptySnaptb-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307631586 type: DISABLED version: 2 ttl: 0 ) in 49 msec 2023-02-13 17:00:45,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-13 17:00:45,053 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676307630447, procId: 119 completed 2023-02-13 17:00:45,053 INFO [Listener at localhost/40167] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676307630447-1676307644528 2023-02-13 17:00:45,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1676307630447-1676307644528" 2023-02-13 17:00:45,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676307630447-1676307644528 2023-02-13 17:00:45,057 INFO [Listener at localhost/40167] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:45,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676307630447 2023-02-13 17:00:45,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:45,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 17:00:45,060 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676307630447 2023-02-13 17:00:45,060 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307645060"}]},"ts":"1676307645060"} 2023-02-13 17:00:45,062 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLING in hbase:meta 2023-02-13 17:00:45,064 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 17:00:45,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN}] 2023-02-13 17:00:45,065 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN 2023-02-13 17:00:45,066 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN 2023-02-13 17:00:45,066 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN 2023-02-13 17:00:45,066 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN 2023-02-13 17:00:45,066 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN 2023-02-13 17:00:45,066 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:45,066 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:45,067 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:45,067 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:45,067 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:45,067 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN 2023-02-13 17:00:45,068 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:45,217 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:00:45,217 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:00:45,217 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:00:45,217 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:00:45,217 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:00:45,221 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:45,221 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:45,221 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:45,222 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307645221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645221"}]},"ts":"1676307645221"} 2023-02-13 17:00:45,221 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:45,221 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:45,222 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645221"}]},"ts":"1676307645221"} 2023-02-13 17:00:45,222 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645221"}]},"ts":"1676307645221"} 2023-02-13 17:00:45,222 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307645221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645221"}]},"ts":"1676307645221"} 2023-02-13 17:00:45,222 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645221"}]},"ts":"1676307645221"} 2023-02-13 17:00:45,223 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=126, state=RUNNABLE; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:45,224 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=125, state=RUNNABLE; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:45,225 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=122, state=RUNNABLE; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:45,226 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=121, state=RUNNABLE; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:45,227 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:45,227 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=124, state=RUNNABLE; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:45,227 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645227"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645227"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645227"}]},"ts":"1676307645227"} 2023-02-13 17:00:45,228 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=123, state=RUNNABLE; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:45,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 17:00:45,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:45,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 17:00:45,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:45,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:45,378 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:45,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:45,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 17:00:45,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:45,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:45,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:45,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:45,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:45,379 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:45,380 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:45,380 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:45,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 17:00:45,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:45,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:45,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:45,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:45,381 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:45,381 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:45,381 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:45,381 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:45,381 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61523f42895e23c1110bbb6a6531b0a6 columnFamilyName cf 2023-02-13 17:00:45,382 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7925c8f572739336270bf824c12623ea columnFamilyName cf 2023-02-13 17:00:45,383 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:45,383 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(310): Store=7925c8f572739336270bf824c12623ea/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:45,383 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(310): Store=61523f42895e23c1110bbb6a6531b0a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:45,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:45,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:45,384 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:45,384 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:45,384 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f31ed543f552e93833ffd6a17e17334b columnFamilyName cf 2023-02-13 17:00:45,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:45,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:45,385 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(310): Store=f31ed543f552e93833ffd6a17e17334b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:45,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:45,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:45,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:45,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:45,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 61523f42895e23c1110bbb6a6531b0a6; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73979089, jitterRate=0.10237433016300201}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:45,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 61523f42895e23c1110bbb6a6531b0a6: Running coprocessor pre-open hook at 1676307645379Writing region info on filesystem at 1676307645379Initializing all the Stores at 1676307645379Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307645379Cleaning up temporary data from old regions at 1676307645385 (+6 ms)Cleaning up detritus from prior splits at 1676307645385Running coprocessor post-open hooks at 1676307645388 (+3 ms)Region opened successfully at 1676307645388 2023-02-13 17:00:45,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7925c8f572739336270bf824c12623ea; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67881420, jitterRate=0.01151198148727417}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:45,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7925c8f572739336270bf824c12623ea: Running coprocessor pre-open hook at 1676307645379Writing region info on filesystem at 1676307645379Initializing all the Stores at 1676307645379Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307645379Cleaning up temporary data from old regions at 1676307645385 (+6 ms)Cleaning up detritus from prior splits at 1676307645385Running coprocessor post-open hooks at 1676307645388 (+3 ms)Region opened successfully at 1676307645388 2023-02-13 17:00:45,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6., pid=127, masterSystemTime=1676307645375 2023-02-13 17:00:45,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea., pid=128, masterSystemTime=1676307645376 2023-02-13 17:00:45,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:45,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f31ed543f552e93833ffd6a17e17334b; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64180940, jitterRate=-0.043629467487335205}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:45,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f31ed543f552e93833ffd6a17e17334b: Running coprocessor pre-open hook at 1676307645381Writing region info on filesystem at 1676307645381Initializing all the Stores at 1676307645382 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307645382Cleaning up temporary data from old regions at 1676307645387 (+5 ms)Cleaning up detritus from prior splits at 1676307645388 (+1 ms)Running coprocessor post-open hooks at 1676307645390 (+2 ms)Region opened successfully at 1676307645390 2023-02-13 17:00:45,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:45,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:45,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:45,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 17:00:45,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:45,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b., pid=131, masterSystemTime=1676307645378 2023-02-13 17:00:45,391 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:45,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:45,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:45,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:45,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:45,391 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307645391"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307645391"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307645391"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307645391"}]},"ts":"1676307645391"} 2023-02-13 17:00:45,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:45,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:45,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 17:00:45,391 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:45,392 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645391"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307645391"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307645391"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307645391"}]},"ts":"1676307645391"} 2023-02-13 17:00:45,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:45,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:45,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:45,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:45,393 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:45,393 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:45,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:45,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:45,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:45,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 17:00:45,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:45,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:45,395 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:45,395 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:45,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:45,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:45,395 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2457c5d248f99f39cff76beaa71b4345 columnFamilyName cf 2023-02-13 17:00:45,395 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:45,396 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:45,396 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:45,396 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645395"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307645395"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307645395"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307645395"}]},"ts":"1676307645395"} 2023-02-13 17:00:45,396 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 columnFamilyName cf 2023-02-13 17:00:45,396 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=126 2023-02-13 17:00:45,396 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=126, state=SUCCESS; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,34035,1676307627747 in 170 msec 2023-02-13 17:00:45,397 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(310): Store=bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:45,397 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(310): Store=2457c5d248f99f39cff76beaa71b4345/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:45,397 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:45,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:45,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:45,399 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:45,399 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:45,399 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e16e4423db3157675739f103f8f5a30 columnFamilyName cf 2023-02-13 17:00:45,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:45,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:45,400 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(310): Store=0e16e4423db3157675739f103f8f5a30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:45,400 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=125 2023-02-13 17:00:45,400 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=125, state=SUCCESS; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670 in 170 msec 2023-02-13 17:00:45,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:45,401 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN in 332 msec 2023-02-13 17:00:45,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:45,402 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=124 2023-02-13 17:00:45,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN in 336 msec 2023-02-13 17:00:45,402 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=124, state=SUCCESS; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,43065,1676307627716 in 170 msec 2023-02-13 17:00:45,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:45,403 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bb0e3d66ca9f4f9ce3e87dc8bfc74c30; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70083966, jitterRate=0.04433247447013855}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:45,403 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Running coprocessor pre-open hook at 1676307645392Writing region info on filesystem at 1676307645392Initializing all the Stores at 1676307645393 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307645393Cleaning up temporary data from old regions at 1676307645400 (+7 ms)Cleaning up detritus from prior splits at 1676307645400Running coprocessor post-open hooks at 1676307645403 (+3 ms)Region opened successfully at 1676307645403 2023-02-13 17:00:45,405 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN in 338 msec 2023-02-13 17:00:45,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:45,405 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30., pid=132, masterSystemTime=1676307645375 2023-02-13 17:00:45,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:45,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e16e4423db3157675739f103f8f5a30; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75343751, jitterRate=0.12270937860012054}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:45,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e16e4423db3157675739f103f8f5a30: Running coprocessor pre-open hook at 1676307645395Writing region info on filesystem at 1676307645395Initializing all the Stores at 1676307645397 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307645397Cleaning up temporary data from old regions at 1676307645402 (+5 ms)Cleaning up detritus from prior splits at 1676307645403 (+1 ms)Running coprocessor post-open hooks at 1676307645406 (+3 ms)Region opened successfully at 1676307645406 2023-02-13 17:00:45,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:45,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:45,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2457c5d248f99f39cff76beaa71b4345; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69723117, jitterRate=0.03895540535449982}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:45,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30., pid=130, masterSystemTime=1676307645378 2023-02-13 17:00:45,407 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:45,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2457c5d248f99f39cff76beaa71b4345: Running coprocessor pre-open hook at 1676307645392Writing region info on filesystem at 1676307645392Initializing all the Stores at 1676307645393 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307645393Cleaning up temporary data from old regions at 1676307645400 (+7 ms)Cleaning up detritus from prior splits at 1676307645400Running coprocessor post-open hooks at 1676307645407 (+7 ms)Region opened successfully at 1676307645407 2023-02-13 17:00:45,407 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645407"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307645407"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307645407"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307645407"}]},"ts":"1676307645407"} 2023-02-13 17:00:45,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345., pid=129, masterSystemTime=1676307645376 2023-02-13 17:00:45,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:45,409 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:45,410 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:45,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:45,410 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307645410"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307645410"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307645410"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307645410"}]},"ts":"1676307645410"} 2023-02-13 17:00:45,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:45,411 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:45,411 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645410"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307645410"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307645410"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307645410"}]},"ts":"1676307645410"} 2023-02-13 17:00:45,413 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=123 2023-02-13 17:00:45,413 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=123, state=SUCCESS; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 181 msec 2023-02-13 17:00:45,414 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=121 2023-02-13 17:00:45,415 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=121, state=SUCCESS; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 186 msec 2023-02-13 17:00:45,415 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN in 349 msec 2023-02-13 17:00:45,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=122 2023-02-13 17:00:45,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=122, state=SUCCESS; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670 in 188 msec 2023-02-13 17:00:45,417 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN in 351 msec 2023-02-13 17:00:45,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=122, resume processing ppid=120 2023-02-13 17:00:45,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN in 351 msec 2023-02-13 17:00:45,419 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307645418"}]},"ts":"1676307645418"} 2023-02-13 17:00:45,420 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLED in hbase:meta 2023-02-13 17:00:45,423 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676307630447' was successfully enabled. 2023-02-13 17:00:45,425 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676307630447 in 367 msec 2023-02-13 17:00:45,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 17:00:45,812 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 120 completed 2023-02-13 17:00:45,812 DEBUG [Listener at localhost/40167] zookeeper.ReadOnlyZKClient(139): Connect 0x58227bee to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:45,818 DEBUG [Listener at localhost/40167] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a24f0e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:45,820 DEBUG [hconnection-0x231aba10-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:45,823 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55246, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:45,827 DEBUG [hconnection-0x231aba10-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:45,829 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39398, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:45,830 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 17:00:45,832 DEBUG [hconnection-0x231aba10-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:45,836 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56796, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:45,837 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 17:00:45,843 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 17:00:45,847 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 17:00:45,851 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 17:00:45,858 INFO [Listener at localhost/40167] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:45,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676307630447 2023-02-13 17:00:45,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:45,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 17:00:45,862 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307645862"}]},"ts":"1676307645862"} 2023-02-13 17:00:45,863 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLING in hbase:meta 2023-02-13 17:00:45,865 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676307630447 to state=DISABLING 2023-02-13 17:00:45,866 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN}] 2023-02-13 17:00:45,868 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN 2023-02-13 17:00:45,868 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN 2023-02-13 17:00:45,868 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN 2023-02-13 17:00:45,868 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN 2023-02-13 17:00:45,868 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN 2023-02-13 17:00:45,868 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:45,868 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645868"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645868"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645868"}]},"ts":"1676307645868"} 2023-02-13 17:00:45,868 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:45,869 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:45,869 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645868"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645868"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645868"}]},"ts":"1676307645868"} 2023-02-13 17:00:45,869 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307645868"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645868"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645868"}]},"ts":"1676307645868"} 2023-02-13 17:00:45,869 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:45,869 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645869"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645869"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645869"}]},"ts":"1676307645869"} 2023-02-13 17:00:45,869 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:45,869 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307645869"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645869"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645869"}]},"ts":"1676307645869"} 2023-02-13 17:00:45,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=138, state=RUNNABLE; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:45,870 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=135, state=RUNNABLE; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:45,871 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=134, state=RUNNABLE; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:45,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=137, state=RUNNABLE; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:45,875 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=139, state=RUNNABLE; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:45,876 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN 2023-02-13 17:00:45,876 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:45,877 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307645876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307645876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307645876"}]},"ts":"1676307645876"} 2023-02-13 17:00:45,878 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=136, state=RUNNABLE; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:46,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:46,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2457c5d248f99f39cff76beaa71b4345, disabling compactions & flushes 2023-02-13 17:00:46,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:46,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:46,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. after waiting 0 ms 2023-02-13 17:00:46,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:46,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:46,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e16e4423db3157675739f103f8f5a30, disabling compactions & flushes 2023-02-13 17:00:46,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:46,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:46,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. after waiting 0 ms 2023-02-13 17:00:46,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:46,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:46,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 17:00:46,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 61523f42895e23c1110bbb6a6531b0a6, disabling compactions & flushes 2023-02-13 17:00:46,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:46,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:46,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. after waiting 0 ms 2023-02-13 17:00:46,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:46,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2457c5d248f99f39cff76beaa71b4345: Waiting for close lock at 1676307646022Running coprocessor pre-close hooks at 1676307646022Disabling compacts and flushes for region at 1676307646022Disabling writes for close at 1676307646023 (+1 ms)Writing region close event to WAL at 1676307646024 (+1 ms)Running coprocessor post-close hooks at 1676307646030 (+6 ms)Closed at 1676307646030 2023-02-13 17:00:46,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:46,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:46,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7925c8f572739336270bf824c12623ea, disabling compactions & flushes 2023-02-13 17:00:46,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:46,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:46,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. after waiting 0 ms 2023-02-13 17:00:46,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:46,033 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSED 2023-02-13 17:00:46,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 17:00:46,033 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646033"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646033"}]},"ts":"1676307646033"} 2023-02-13 17:00:46,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:46,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e16e4423db3157675739f103f8f5a30: Waiting for close lock at 1676307646026Running coprocessor pre-close hooks at 1676307646026Disabling compacts and flushes for region at 1676307646026Disabling writes for close at 1676307646027 (+1 ms)Writing region close event to WAL at 1676307646028 (+1 ms)Running coprocessor post-close hooks at 1676307646033 (+5 ms)Closed at 1676307646033 2023-02-13 17:00:46,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 17:00:46,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:46,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 61523f42895e23c1110bbb6a6531b0a6: Waiting for close lock at 1676307646028Running coprocessor pre-close hooks at 1676307646028Disabling compacts and flushes for region at 1676307646028Disabling writes for close at 1676307646029 (+1 ms)Writing region close event to WAL at 1676307646030 (+1 ms)Running coprocessor post-close hooks at 1676307646034 (+4 ms)Closed at 1676307646034 2023-02-13 17:00:46,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:46,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:46,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f31ed543f552e93833ffd6a17e17334b, disabling compactions & flushes 2023-02-13 17:00:46,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:46,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:46,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. after waiting 0 ms 2023-02-13 17:00:46,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:46,036 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSED 2023-02-13 17:00:46,037 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307646036"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646036"}]},"ts":"1676307646036"} 2023-02-13 17:00:46,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:46,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:46,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bb0e3d66ca9f4f9ce3e87dc8bfc74c30, disabling compactions & flushes 2023-02-13 17:00:46,038 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:46,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 17:00:46,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:46,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. after waiting 0 ms 2023-02-13 17:00:46,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:46,039 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSED 2023-02-13 17:00:46,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:46,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7925c8f572739336270bf824c12623ea: Waiting for close lock at 1676307646031Running coprocessor pre-close hooks at 1676307646031Disabling compacts and flushes for region at 1676307646031Disabling writes for close at 1676307646032 (+1 ms)Writing region close event to WAL at 1676307646033 (+1 ms)Running coprocessor post-close hooks at 1676307646039 (+6 ms)Closed at 1676307646039 2023-02-13 17:00:46,039 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307646039"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646039"}]},"ts":"1676307646039"} 2023-02-13 17:00:46,039 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=135 2023-02-13 17:00:46,039 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=135, state=SUCCESS; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670 in 165 msec 2023-02-13 17:00:46,041 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:46,041 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 17:00:46,041 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSED 2023-02-13 17:00:46,041 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN in 173 msec 2023-02-13 17:00:46,041 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646041"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646041"}]},"ts":"1676307646041"} 2023-02-13 17:00:46,042 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:46,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f31ed543f552e93833ffd6a17e17334b: Waiting for close lock at 1676307646035Running coprocessor pre-close hooks at 1676307646035Disabling compacts and flushes for region at 1676307646035Disabling writes for close at 1676307646036 (+1 ms)Writing region close event to WAL at 1676307646036Running coprocessor post-close hooks at 1676307646042 (+6 ms)Closed at 1676307646042 2023-02-13 17:00:46,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 17:00:46,043 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=134 2023-02-13 17:00:46,043 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=134, state=SUCCESS; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 168 msec 2023-02-13 17:00:46,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:46,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Waiting for close lock at 1676307646037Running coprocessor pre-close hooks at 1676307646037Disabling compacts and flushes for region at 1676307646037Disabling writes for close at 1676307646038 (+1 ms)Writing region close event to WAL at 1676307646039 (+1 ms)Running coprocessor post-close hooks at 1676307646043 (+4 ms)Closed at 1676307646043 2023-02-13 17:00:46,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:46,044 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=139 2023-02-13 17:00:46,044 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN in 177 msec 2023-02-13 17:00:46,044 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=139, state=SUCCESS; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,34035,1676307627747 in 166 msec 2023-02-13 17:00:46,044 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSED 2023-02-13 17:00:46,045 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646044"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646044"}]},"ts":"1676307646044"} 2023-02-13 17:00:46,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:46,045 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSED 2023-02-13 17:00:46,045 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646045"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646045"}]},"ts":"1676307646045"} 2023-02-13 17:00:46,046 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN in 178 msec 2023-02-13 17:00:46,047 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=138 2023-02-13 17:00:46,047 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=138, state=SUCCESS; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670 in 174 msec 2023-02-13 17:00:46,048 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=137 2023-02-13 17:00:46,048 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN in 181 msec 2023-02-13 17:00:46,049 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=137, state=SUCCESS; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,43065,1676307627716 in 172 msec 2023-02-13 17:00:46,049 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=136 2023-02-13 17:00:46,049 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=136, state=SUCCESS; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 169 msec 2023-02-13 17:00:46,050 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN in 182 msec 2023-02-13 17:00:46,051 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=136, resume processing ppid=133 2023-02-13 17:00:46,051 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN in 183 msec 2023-02-13 17:00:46,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307646052"}]},"ts":"1676307646052"} 2023-02-13 17:00:46,053 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLED in hbase:meta 2023-02-13 17:00:46,055 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676307630447 to state=DISABLED 2023-02-13 17:00:46,057 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676307630447 in 197 msec 2023-02-13 17:00:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 17:00:46,113 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 133 completed 2023-02-13 17:00:46,123 INFO [Listener at localhost/40167] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676307630447-1676307646123 2023-02-13 17:00:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676307630447-1676307646123 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:46,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676307646124 (current time:1676307646124). 2023-02-13 17:00:46,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 17:00:46,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(139): Connect 0x67a44826 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:46,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ed30414, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:46,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:46,134 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55258, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:46,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67a44826 to 127.0.0.1:54083 2023-02-13 17:00:46,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:46,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 17:00:46,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676307630447-1676307646123 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:46,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:46,138 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:46,139 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE; timeout=600000, timestamp=1676308246139 2023-02-13 17:00:46,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676307630447-1676307646123 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } 2023-02-13 17:00:46,139 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676307630447-1676307646123 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,139 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676307630447-1676307646123 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 17:00:46,139 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 17:00:46,148 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676307630447-1676307646123 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } 2023-02-13 17:00:46,149 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:46,149 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:46,149 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:46,150 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:46,151 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:46,151 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:46,151 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:46,152 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:46,152 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:46,153 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:46,156 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:46,183 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676307630447-1676307646123 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } as finished. 2023-02-13 17:00:46,183 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676307630447-1676307646123 2023-02-13 17:00:46,184 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676307630447-1676307646123 2023-02-13 17:00:46,209 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676307630447-1676307646123 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/hbase-failsafe-snaptb1-1676307630447-1676307646123 2023-02-13 17:00:46,227 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676307630447-1676307646123 of table testRestoreSnapshot-1676307630447 completed 2023-02-13 17:00:46,228 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676307630447-1676307646123 2023-02-13 17:00:46,228 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676307630447-1676307646123 2023-02-13 17:00:46,228 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676307630447-1676307646123 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676307630447 at 1676307646139Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676307630447-1676307646123 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } at 1676307646148 (+9 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676307630447-1676307646123 at 1676307646183 (+35 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676307630447-1676307646123 at 1676307646183Writing data manifest for hbase-failsafe-snaptb1-1676307630447-1676307646123 at 1676307646192 (+9 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676307630447-1676307646123 at 1676307646201 (+9 ms)Snapshot hbase-failsafe-snaptb1-1676307630447-1676307646123 of table testRestoreSnapshot-1676307630447 completed at 1676307646227 (+26 ms) 2023-02-13 17:00:46,230 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE 2023-02-13 17:00:46,231 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676307630447, type=EXCLUSIVE in 93 msec 2023-02-13 17:00:46,389 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 17:00:46,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676307630447-1676307646123 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 } is done 2023-02-13 17:00:46,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676307630447-1676307646123 table=testRestoreSnapshot-1676307630447 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 17:00:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676307630447 snapshot=name: "snaptb1-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307641126 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 17:00:46,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676307630447 as table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-13 17:00:46,418 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000004 2023-02-13 17:00:46,419 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000003 2023-02-13 17:00:46,419 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000004 2023-02-13 17:00:46,420 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } 2023-02-13 17:00:46,423 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307641126 type: DISABLED version: 2 ttl: 0 2023-02-13 17:00:46,423 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,430 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,431 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,435 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,438 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,441 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,444 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,447 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,451 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,454 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,457 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,460 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,464 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,467 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 of snapshot=snaptb1-1676307630447 to region=7469c224c9ad52a75eb8915b71d139b8 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,471 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:46,471 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:46,471 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:46,471 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:46,471 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:46,471 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:46,473 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 47c584031cad45b68b3da5b7c3330a2d of snapshot=snaptb1-1676307630447 to region=0e16e4423db3157675739f103f8f5a30 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,473 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 15eaf8061d754363820c5e36fdb4956a of snapshot=snaptb1-1676307630447 to region=2457c5d248f99f39cff76beaa71b4345 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,475 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 52130edd005d4ce89422019dd11d135a of snapshot=snaptb1-1676307630447 to region=7925c8f572739336270bf824c12623ea table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,475 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 5f02a25012d1463cb5f37ea13b2cf2dd of snapshot=snaptb1-1676307630447 to region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,475 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 1b81c71679994dda95fb56406c0c1e6d of snapshot=snaptb1-1676307630447 to region=f31ed543f552e93833ffd6a17e17334b table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,476 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 522b6eff1a16428ab53f7cfcc13f5481 of snapshot=snaptb1-1676307630447 to region=61523f42895e23c1110bbb6a6531b0a6 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,484 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f6d566fa52ad45b6ba62c7872adbae6b of snapshot=snaptb1-1676307630447 to region=0e16e4423db3157675739f103f8f5a30 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,484 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 97a455c0271c489d81c6c7b49b2cf4d9 of snapshot=snaptb1-1676307630447 to region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,484 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f6bae3d5c579413fb7516ba3d60e58f8 of snapshot=snaptb1-1676307630447 to region=61523f42895e23c1110bbb6a6531b0a6 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,484 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 46cd66f26a1c432184d8807fe83439c0 of snapshot=snaptb1-1676307630447 to region=2457c5d248f99f39cff76beaa71b4345 table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,484 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 7909cb0a0160488495640804774da893 of snapshot=snaptb1-1676307630447 to region=f31ed543f552e93833ffd6a17e17334b table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,484 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d4a69b191ab64507915a551f7d3cf0ec of snapshot=snaptb1-1676307630447 to region=7925c8f572739336270bf824c12623ea table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,493 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307641126 type: DISABLED version: 2 ttl: 0 2023-02-13 17:00:46,494 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307646494"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:46,494 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307646494"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:46,494 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307646494"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:46,494 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307646494"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:46,494 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307646494"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:46,494 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307646494"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:46,496 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 17:00:46,496 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 17:00:46,496 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307646495"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646495"}]},"ts":"1676307646495"} 2023-02-13 17:00:46,496 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646495"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646495"}]},"ts":"1676307646495"} 2023-02-13 17:00:46,496 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307646495"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646495"}]},"ts":"1676307646495"} 2023-02-13 17:00:46,496 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646495"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646495"}]},"ts":"1676307646495"} 2023-02-13 17:00:46,496 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646495"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646495"}]},"ts":"1676307646495"} 2023-02-13 17:00:46,496 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646495"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307646495"}]},"ts":"1676307646495"} 2023-02-13 17:00:46,498 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 17:00:46,498 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 17:00:46,498 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 17:00:46,498 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676307630447 table=testRestoreSnapshot-1676307630447 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676307630447 completed! 2023-02-13 17:00:46,500 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676307630447 snapshot=name: "snaptb1-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307641126 type: DISABLED version: 2 ttl: 0 ) in 96 msec 2023-02-13 17:00:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-13 17:00:46,660 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676307630447, procId: 147 completed 2023-02-13 17:00:46,660 INFO [Listener at localhost/40167] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676307630447-1676307646123 2023-02-13 17:00:46,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1676307630447-1676307646123" 2023-02-13 17:00:46,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676307630447-1676307646123 2023-02-13 17:00:46,663 INFO [Listener at localhost/40167] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 17:00:46,667 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676307630447 2023-02-13 17:00:46,667 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307646667"}]},"ts":"1676307646667"} 2023-02-13 17:00:46,668 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLING in hbase:meta 2023-02-13 17:00:46,671 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 17:00:46,671 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN}] 2023-02-13 17:00:46,673 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN 2023-02-13 17:00:46,673 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN 2023-02-13 17:00:46,673 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN 2023-02-13 17:00:46,674 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN 2023-02-13 17:00:46,674 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:46,674 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:46,675 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:46,675 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:46,675 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN 2023-02-13 17:00:46,675 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN 2023-02-13 17:00:46,676 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:46,676 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 17:00:46,824 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:00:46,824 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:00:46,824 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:00:46,824 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:00:46,824 DEBUG [jenkins-hbase4:40255] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:00:46,828 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:46,828 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:46,828 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646827"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307646827"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307646827"}]},"ts":"1676307646827"} 2023-02-13 17:00:46,828 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307646828"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307646828"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307646828"}]},"ts":"1676307646828"} 2023-02-13 17:00:46,828 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:46,828 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:46,828 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646828"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307646828"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307646828"}]},"ts":"1676307646828"} 2023-02-13 17:00:46,828 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646827"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307646827"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307646827"}]},"ts":"1676307646827"} 2023-02-13 17:00:46,828 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:46,828 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307646828"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307646828"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307646828"}]},"ts":"1676307646828"} 2023-02-13 17:00:46,829 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=153, state=RUNNABLE; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:46,830 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=152, state=RUNNABLE; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:46,831 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=150, state=RUNNABLE; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:46,832 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=149, state=RUNNABLE; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:46,832 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:46,832 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307646832"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307646832"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307646832"}]},"ts":"1676307646832"} 2023-02-13 17:00:46,833 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=154, state=RUNNABLE; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:46,834 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=151, state=RUNNABLE; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:46,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 17:00:46,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:46,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 17:00:46,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:46,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:46,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:46,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 17:00:46,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:46,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:46,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:46,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:46,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:46,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:46,986 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:46,986 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:46,986 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:46,987 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:46,986 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:46,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 17:00:46,987 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:46,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:46,987 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7925c8f572739336270bf824c12623ea columnFamilyName cf 2023-02-13 17:00:46,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:46,987 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:46,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:46,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:46,988 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e16e4423db3157675739f103f8f5a30 columnFamilyName cf 2023-02-13 17:00:46,988 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:46,989 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:46,989 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:46,989 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 columnFamilyName cf 2023-02-13 17:00:47,005 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:47,005 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-5f02a25012d1463cb5f37ea13b2cf2dd 2023-02-13 17:00:47,005 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:47,013 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-f6d566fa52ad45b6ba62c7872adbae6b 2023-02-13 17:00:47,013 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(310): Store=0e16e4423db3157675739f103f8f5a30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:47,014 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:47,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:47,014 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(310): Store=bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:47,014 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-d4a69b191ab64507915a551f7d3cf0ec 2023-02-13 17:00:47,015 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(310): Store=7925c8f572739336270bf824c12623ea/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:47,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:47,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:47,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:47,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:47,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:47,019 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:47,019 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e16e4423db3157675739f103f8f5a30; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59932444, jitterRate=-0.1069369912147522}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:47,019 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e16e4423db3157675739f103f8f5a30: Running coprocessor pre-open hook at 1676307646985Writing region info on filesystem at 1676307646985Initializing all the Stores at 1676307646986 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307646986Cleaning up temporary data from old regions at 1676307647016 (+30 ms)Cleaning up detritus from prior splits at 1676307647016Running coprocessor post-open hooks at 1676307647019 (+3 ms)Region opened successfully at 1676307647019 2023-02-13 17:00:47,019 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7925c8f572739336270bf824c12623ea; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63271817, jitterRate=-0.05717645585536957}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:47,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7925c8f572739336270bf824c12623ea: Running coprocessor pre-open hook at 1676307646985Writing region info on filesystem at 1676307646985Initializing all the Stores at 1676307646985Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307646985Cleaning up temporary data from old regions at 1676307647017 (+32 ms)Cleaning up detritus from prior splits at 1676307647017Running coprocessor post-open hooks at 1676307647020 (+3 ms)Region opened successfully at 1676307647020 2023-02-13 17:00:47,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:47,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30., pid=158, masterSystemTime=1676307646982 2023-02-13 17:00:47,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea., pid=155, masterSystemTime=1676307646981 2023-02-13 17:00:47,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:47,022 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:47,022 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:47,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 17:00:47,022 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:47,023 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307647022"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307647022"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307647022"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307647022"}]},"ts":"1676307647022"} 2023-02-13 17:00:47,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:47,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:47,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:47,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:47,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:47,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:47,024 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:47,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 17:00:47,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:47,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:47,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:47,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:47,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:47,024 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:47,025 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:47,025 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307647025"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307647025"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307647025"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307647025"}]},"ts":"1676307647025"} 2023-02-13 17:00:47,026 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:47,026 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:47,026 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bb0e3d66ca9f4f9ce3e87dc8bfc74c30; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74080900, jitterRate=0.10389143228530884}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:47,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Running coprocessor pre-open hook at 1676307646987Writing region info on filesystem at 1676307646987Initializing all the Stores at 1676307646988 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307646988Cleaning up temporary data from old regions at 1676307647020 (+32 ms)Cleaning up detritus from prior splits at 1676307647021 (+1 ms)Running coprocessor post-open hooks at 1676307647026 (+5 ms)Region opened successfully at 1676307647026 2023-02-13 17:00:47,026 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f31ed543f552e93833ffd6a17e17334b columnFamilyName cf 2023-02-13 17:00:47,026 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:47,027 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=149 2023-02-13 17:00:47,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30., pid=160, masterSystemTime=1676307646984 2023-02-13 17:00:47,027 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=149, state=SUCCESS; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 193 msec 2023-02-13 17:00:47,028 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:47,028 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:47,028 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2457c5d248f99f39cff76beaa71b4345 columnFamilyName cf 2023-02-13 17:00:47,029 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN in 356 msec 2023-02-13 17:00:47,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:47,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:47,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:47,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 17:00:47,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:47,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:47,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:47,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:47,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=153 2023-02-13 17:00:47,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=153, state=SUCCESS; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670 in 198 msec 2023-02-13 17:00:47,030 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:47,031 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307647030"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307647030"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307647030"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307647030"}]},"ts":"1676307647030"} 2023-02-13 17:00:47,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN in 359 msec 2023-02-13 17:00:47,033 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=151 2023-02-13 17:00:47,033 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=151, state=SUCCESS; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 198 msec 2023-02-13 17:00:47,035 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN in 362 msec 2023-02-13 17:00:47,039 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:47,040 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:47,040 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:47,041 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61523f42895e23c1110bbb6a6531b0a6 columnFamilyName cf 2023-02-13 17:00:47,048 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:47,053 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:47,054 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:47,065 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-7909cb0a0160488495640804774da893 2023-02-13 17:00:47,066 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(310): Store=f31ed543f552e93833ffd6a17e17334b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:47,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:47,068 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:47,069 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-46cd66f26a1c432184d8807fe83439c0 2023-02-13 17:00:47,070 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(310): Store=2457c5d248f99f39cff76beaa71b4345/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:47,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:47,071 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-f6bae3d5c579413fb7516ba3d60e58f8 2023-02-13 17:00:47,071 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(310): Store=61523f42895e23c1110bbb6a6531b0a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:47,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:47,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:47,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:47,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:47,073 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f31ed543f552e93833ffd6a17e17334b; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65037388, jitterRate=-0.030867397785186768}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:47,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f31ed543f552e93833ffd6a17e17334b: Running coprocessor pre-open hook at 1676307647023Writing region info on filesystem at 1676307647023Initializing all the Stores at 1676307647024 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307647024Cleaning up temporary data from old regions at 1676307647069 (+45 ms)Cleaning up detritus from prior splits at 1676307647070 (+1 ms)Running coprocessor post-open hooks at 1676307647074 (+4 ms)Region opened successfully at 1676307647074 2023-02-13 17:00:47,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b., pid=156, masterSystemTime=1676307646982 2023-02-13 17:00:47,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:47,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2457c5d248f99f39cff76beaa71b4345; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73738899, jitterRate=0.09879522025585175}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:47,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2457c5d248f99f39cff76beaa71b4345: Running coprocessor pre-open hook at 1676307647025Writing region info on filesystem at 1676307647025Initializing all the Stores at 1676307647026 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307647026Cleaning up temporary data from old regions at 1676307647072 (+46 ms)Cleaning up detritus from prior splits at 1676307647073 (+1 ms)Running coprocessor post-open hooks at 1676307647076 (+3 ms)Region opened successfully at 1676307647076 2023-02-13 17:00:47,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:47,077 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:47,077 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:47,077 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345., pid=157, masterSystemTime=1676307646981 2023-02-13 17:00:47,077 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307647077"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307647077"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307647077"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307647077"}]},"ts":"1676307647077"} 2023-02-13 17:00:47,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:47,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:47,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 61523f42895e23c1110bbb6a6531b0a6; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71040268, jitterRate=0.05858248472213745}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:47,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:47,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 61523f42895e23c1110bbb6a6531b0a6: Running coprocessor pre-open hook at 1676307647030Writing region info on filesystem at 1676307647030Initializing all the Stores at 1676307647031 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307647031Cleaning up temporary data from old regions at 1676307647074 (+43 ms)Cleaning up detritus from prior splits at 1676307647074Running coprocessor post-open hooks at 1676307647078 (+4 ms)Region opened successfully at 1676307647078 2023-02-13 17:00:47,079 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:47,079 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307647079"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307647079"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307647079"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307647079"}]},"ts":"1676307647079"} 2023-02-13 17:00:47,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6., pid=159, masterSystemTime=1676307646984 2023-02-13 17:00:47,080 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:47,081 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:47,081 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=152 2023-02-13 17:00:47,081 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=152, state=SUCCESS; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,43065,1676307627716 in 248 msec 2023-02-13 17:00:47,081 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:47,082 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307647081"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307647081"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307647081"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307647081"}]},"ts":"1676307647081"} 2023-02-13 17:00:47,084 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=150 2023-02-13 17:00:47,084 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=150, state=SUCCESS; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670 in 250 msec 2023-02-13 17:00:47,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN in 411 msec 2023-02-13 17:00:47,085 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=154 2023-02-13 17:00:47,085 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN in 413 msec 2023-02-13 17:00:47,085 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=154, state=SUCCESS; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,34035,1676307627747 in 250 msec 2023-02-13 17:00:47,086 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=154, resume processing ppid=148 2023-02-13 17:00:47,086 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN in 414 msec 2023-02-13 17:00:47,087 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307647087"}]},"ts":"1676307647087"} 2023-02-13 17:00:47,088 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLED in hbase:meta 2023-02-13 17:00:47,090 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676307630447' was successfully enabled. 2023-02-13 17:00:47,091 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676307630447 in 426 msec 2023-02-13 17:00:47,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 17:00:47,418 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 148 completed 2023-02-13 17:00:47,419 DEBUG [Listener at localhost/40167] zookeeper.ReadOnlyZKClient(139): Connect 0x131d42f3 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:47,424 DEBUG [Listener at localhost/40167] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29616110, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:47,426 DEBUG [hconnection-0x49c92e86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:47,428 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55272, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:47,431 DEBUG [hconnection-0x49c92e86-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:47,433 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39410, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:47,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,656 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 17:00:47,660 DEBUG [hconnection-0x49c92e86-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:47,661 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56812, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:47,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,799 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 17:00:47,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,831 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e907963 to 127.0.0.1:54083 2023-02-13 17:00:47,831 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:47,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,832 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58227bee to 127.0.0.1:54083 2023-02-13 17:00:47,832 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:47,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,944 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 17:00:47,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:47,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,135 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 17:00:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,303 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 17:00:48,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,516 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 17:00:48,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:48,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:49,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:50,624 INFO [Listener at localhost/40167] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:50,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676307630447 2023-02-13 17:00:50,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:50,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 17:00:50,631 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307650630"}]},"ts":"1676307650630"} 2023-02-13 17:00:50,632 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLING in hbase:meta 2023-02-13 17:00:50,634 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676307630447 to state=DISABLING 2023-02-13 17:00:50,634 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN}] 2023-02-13 17:00:50,636 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN 2023-02-13 17:00:50,636 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN 2023-02-13 17:00:50,637 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN 2023-02-13 17:00:50,637 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN 2023-02-13 17:00:50,637 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:50,637 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN 2023-02-13 17:00:50,637 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307650637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307650637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307650637"}]},"ts":"1676307650637"} 2023-02-13 17:00:50,638 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:50,638 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:50,638 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307650638"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307650638"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307650638"}]},"ts":"1676307650638"} 2023-02-13 17:00:50,638 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307650638"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307650638"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307650638"}]},"ts":"1676307650638"} 2023-02-13 17:00:50,638 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:50,638 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:50,638 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307650638"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307650638"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307650638"}]},"ts":"1676307650638"} 2023-02-13 17:00:50,638 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307650638"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307650638"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307650638"}]},"ts":"1676307650638"} 2023-02-13 17:00:50,638 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=163, state=RUNNABLE; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:50,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=162, state=RUNNABLE; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:50,640 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=165, state=RUNNABLE; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:50,642 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=164, state=RUNNABLE; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:50,642 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN 2023-02-13 17:00:50,642 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=167, state=RUNNABLE; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:50,643 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:50,643 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307650643"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307650643"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307650643"}]},"ts":"1676307650643"} 2023-02-13 17:00:50,644 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=166, state=RUNNABLE; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:50,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:50,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2457c5d248f99f39cff76beaa71b4345, disabling compactions & flushes 2023-02-13 17:00:50,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:50,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:50,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. after waiting 0 ms 2023-02-13 17:00:50,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:50,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:50,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e16e4423db3157675739f103f8f5a30, disabling compactions & flushes 2023-02-13 17:00:50,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:50,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:50,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. after waiting 0 ms 2023-02-13 17:00:50,795 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:50,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:50,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 61523f42895e23c1110bbb6a6531b0a6, disabling compactions & flushes 2023-02-13 17:00:50,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:50,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:50,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. after waiting 0 ms 2023-02-13 17:00:50,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:50,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 17:00:50,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 17:00:50,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 17:00:50,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:50,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:50,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2457c5d248f99f39cff76beaa71b4345: Waiting for close lock at 1676307650791Running coprocessor pre-close hooks at 1676307650791Disabling compacts and flushes for region at 1676307650791Disabling writes for close at 1676307650792 (+1 ms)Writing region close event to WAL at 1676307650796 (+4 ms)Running coprocessor post-close hooks at 1676307650806 (+10 ms)Closed at 1676307650806 2023-02-13 17:00:50,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e16e4423db3157675739f103f8f5a30: Waiting for close lock at 1676307650792Running coprocessor pre-close hooks at 1676307650792Disabling compacts and flushes for region at 1676307650792Disabling writes for close at 1676307650795 (+3 ms)Writing region close event to WAL at 1676307650799 (+4 ms)Running coprocessor post-close hooks at 1676307650806 (+7 ms)Closed at 1676307650807 (+1 ms) 2023-02-13 17:00:50,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:50,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 61523f42895e23c1110bbb6a6531b0a6: Waiting for close lock at 1676307650796Running coprocessor pre-close hooks at 1676307650796Disabling compacts and flushes for region at 1676307650796Disabling writes for close at 1676307650798 (+2 ms)Writing region close event to WAL at 1676307650802 (+4 ms)Running coprocessor post-close hooks at 1676307650807 (+5 ms)Closed at 1676307650807 2023-02-13 17:00:50,809 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:50,809 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:50,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7925c8f572739336270bf824c12623ea, disabling compactions & flushes 2023-02-13 17:00:50,809 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:50,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:50,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. after waiting 0 ms 2023-02-13 17:00:50,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:50,809 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSED 2023-02-13 17:00:50,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:50,810 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307650809"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307650809"}]},"ts":"1676307650809"} 2023-02-13 17:00:50,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:50,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f31ed543f552e93833ffd6a17e17334b, disabling compactions & flushes 2023-02-13 17:00:50,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:50,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:50,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. after waiting 0 ms 2023-02-13 17:00:50,812 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSED 2023-02-13 17:00:50,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:50,813 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307650812"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307650812"}]},"ts":"1676307650812"} 2023-02-13 17:00:50,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:50,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:50,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bb0e3d66ca9f4f9ce3e87dc8bfc74c30, disabling compactions & flushes 2023-02-13 17:00:50,815 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSED 2023-02-13 17:00:50,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:50,815 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307650814"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307650814"}]},"ts":"1676307650814"} 2023-02-13 17:00:50,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:50,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. after waiting 0 ms 2023-02-13 17:00:50,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:50,820 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=162 2023-02-13 17:00:50,820 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=162, state=SUCCESS; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 179 msec 2023-02-13 17:00:50,821 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=167 2023-02-13 17:00:50,821 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=167, state=SUCCESS; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,34035,1676307627747 in 177 msec 2023-02-13 17:00:50,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN in 186 msec 2023-02-13 17:00:50,822 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=163 2023-02-13 17:00:50,822 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN in 187 msec 2023-02-13 17:00:50,822 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=163, state=SUCCESS; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,43327,1676307627670 in 182 msec 2023-02-13 17:00:50,823 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN in 188 msec 2023-02-13 17:00:50,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 17:00:50,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 17:00:50,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:50,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:50,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f31ed543f552e93833ffd6a17e17334b: Waiting for close lock at 1676307650810Running coprocessor pre-close hooks at 1676307650810Disabling compacts and flushes for region at 1676307650810Disabling writes for close at 1676307650813 (+3 ms)Writing region close event to WAL at 1676307650817 (+4 ms)Running coprocessor post-close hooks at 1676307650827 (+10 ms)Closed at 1676307650828 (+1 ms) 2023-02-13 17:00:50,829 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7925c8f572739336270bf824c12623ea: Waiting for close lock at 1676307650809Running coprocessor pre-close hooks at 1676307650809Disabling compacts and flushes for region at 1676307650809Disabling writes for close at 1676307650809Writing region close event to WAL at 1676307650813 (+4 ms)Running coprocessor post-close hooks at 1676307650828 (+15 ms)Closed at 1676307650828 2023-02-13 17:00:50,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:50,830 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSED 2023-02-13 17:00:50,830 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307650830"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307650830"}]},"ts":"1676307650830"} 2023-02-13 17:00:50,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:50,831 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSED 2023-02-13 17:00:50,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 17:00:50,831 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307650831"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307650831"}]},"ts":"1676307650831"} 2023-02-13 17:00:50,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:50,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Waiting for close lock at 1676307650813Running coprocessor pre-close hooks at 1676307650813Disabling compacts and flushes for region at 1676307650813Disabling writes for close at 1676307650817 (+4 ms)Writing region close event to WAL at 1676307650827 (+10 ms)Running coprocessor post-close hooks at 1676307650831 (+4 ms)Closed at 1676307650832 (+1 ms) 2023-02-13 17:00:50,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:50,838 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSED 2023-02-13 17:00:50,838 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307650838"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307650838"}]},"ts":"1676307650838"} 2023-02-13 17:00:50,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=165 2023-02-13 17:00:50,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=166 2023-02-13 17:00:50,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=165, state=SUCCESS; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,43065,1676307627716 in 192 msec 2023-02-13 17:00:50,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=166, state=SUCCESS; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670 in 189 msec 2023-02-13 17:00:50,840 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN in 205 msec 2023-02-13 17:00:50,840 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN in 205 msec 2023-02-13 17:00:50,841 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=164 2023-02-13 17:00:50,841 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=164, state=SUCCESS; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,34035,1676307627747 in 198 msec 2023-02-13 17:00:50,842 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=164, resume processing ppid=161 2023-02-13 17:00:50,842 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN in 207 msec 2023-02-13 17:00:50,843 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307650843"}]},"ts":"1676307650843"} 2023-02-13 17:00:50,844 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLED in hbase:meta 2023-02-13 17:00:50,847 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676307630447 to state=DISABLED 2023-02-13 17:00:50,848 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676307630447 in 221 msec 2023-02-13 17:00:50,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 17:00:50,882 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 161 completed 2023-02-13 17:00:50,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676307630447 2023-02-13 17:00:50,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:50,894 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:50,895 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:50,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-13 17:00:50,902 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:50,902 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:50,902 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:50,902 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:50,902 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:50,902 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:50,904 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits] 2023-02-13 17:00:50,904 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits] 2023-02-13 17:00:50,904 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits] 2023-02-13 17:00:50,904 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits] 2023-02-13 17:00:50,904 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits] 2023-02-13 17:00:50,905 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits] 2023-02-13 17:00:50,907 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-522b6eff1a16428ab53f7cfcc13f5481 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:50,907 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-15eaf8061d754363820c5e36fdb4956a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:50,908 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-52130edd005d4ce89422019dd11d135a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:50,909 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-47c584031cad45b68b3da5b7c3330a2d to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:50,909 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-1b81c71679994dda95fb56406c0c1e6d to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:50,909 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-5f02a25012d1463cb5f37ea13b2cf2dd to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-5f02a25012d1463cb5f37ea13b2cf2dd 2023-02-13 17:00:50,910 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-f6bae3d5c579413fb7516ba3d60e58f8 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-f6bae3d5c579413fb7516ba3d60e58f8 2023-02-13 17:00:50,910 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-46cd66f26a1c432184d8807fe83439c0 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-46cd66f26a1c432184d8807fe83439c0 2023-02-13 17:00:50,910 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-d4a69b191ab64507915a551f7d3cf0ec to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-d4a69b191ab64507915a551f7d3cf0ec 2023-02-13 17:00:50,911 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-7909cb0a0160488495640804774da893 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-7909cb0a0160488495640804774da893 2023-02-13 17:00:50,911 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-f6d566fa52ad45b6ba62c7872adbae6b to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-f6d566fa52ad45b6ba62c7872adbae6b 2023-02-13 17:00:50,912 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-97a455c0271c489d81c6c7b49b2cf4d9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:50,914 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/27.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/27.seqid 2023-02-13 17:00:50,914 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/27.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/27.seqid 2023-02-13 17:00:50,915 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:50,915 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/27.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/27.seqid 2023-02-13 17:00:50,915 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/27.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/27.seqid 2023-02-13 17:00:50,915 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:50,915 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/27.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/27.seqid 2023-02-13 17:00:50,916 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:50,916 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:50,916 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/27.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/27.seqid 2023-02-13 17:00:50,916 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:50,916 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:50,916 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676307630447 regions 2023-02-13 17:00:50,917 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8 2023-02-13 17:00:50,917 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf] 2023-02-13 17:00:50,919 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a 2023-02-13 17:00:50,920 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 2023-02-13 17:00:50,921 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 2023-02-13 17:00:50,922 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 2023-02-13 17:00:50,923 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 2023-02-13 17:00:50,925 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 2023-02-13 17:00:50,925 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 2023-02-13 17:00:50,926 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 2023-02-13 17:00:50,927 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 2023-02-13 17:00:50,928 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 2023-02-13 17:00:50,929 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 2023-02-13 17:00:50,930 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 2023-02-13 17:00:50,931 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8 2023-02-13 17:00:50,932 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:50,954 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676307630447 from hbase:meta 2023-02-13 17:00:50,956 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676307630447' descriptor. 2023-02-13 17:00:50,957 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:50,957 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676307630447' from region states. 2023-02-13 17:00:50,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307650957"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:50,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307650957"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:50,958 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307650957"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:50,958 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307650957"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:50,958 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307650957"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:50,958 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307650957"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:50,959 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 17:00:50,959 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 17:00:50,959 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676307630447' as deleted. 2023-02-13 17:00:50,960 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676307650960"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:50,961 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676307630447 state from META 2023-02-13 17:00:50,963 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:50,964 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 in 76 msec 2023-02-13 17:00:51,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-13 17:00:51,148 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 174 completed 2023-02-13 17:00:51,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676307630447 snapshot=name: "snaptb1-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307641126 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 17:00:51,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676307630447 as table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 17:00:51,188 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/.tabledesc/.tableinfo.0000000001 2023-02-13 17:00:51,189 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307641126 type: DISABLED version: 2 ttl: 0 2023-02-13 17:00:51,190 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,190 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,192 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,194 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,196 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,198 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,199 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,201 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,203 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,205 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,207 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,209 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,211 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=61523f42895e23c1110bbb6a6531b0a6 as 61523f42895e23c1110bbb6a6531b0a6 in snapshot snaptb1-1676307630447 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=f31ed543f552e93833ffd6a17e17334b as f31ed543f552e93833ffd6a17e17334b in snapshot snaptb1-1676307630447 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=0e16e4423db3157675739f103f8f5a30 as 0e16e4423db3157675739f103f8f5a30 in snapshot snaptb1-1676307630447 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=7925c8f572739336270bf824c12623ea as 7925c8f572739336270bf824c12623ea in snapshot snaptb1-1676307630447 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=2457c5d248f99f39cff76beaa71b4345 as 2457c5d248f99f39cff76beaa71b4345 in snapshot snaptb1-1676307630447 2023-02-13 17:00:51,213 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30 as bb0e3d66ca9f4f9ce3e87dc8bfc74c30 in snapshot snaptb1-1676307630447 2023-02-13 17:00:51,216 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:51,216 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:51,216 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:51,216 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:51,216 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:51,216 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676307630447', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:51,236 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:51,236 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 522b6eff1a16428ab53f7cfcc13f5481 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,239 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f6bae3d5c579413fb7516ba3d60e58f8 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,244 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing 61523f42895e23c1110bbb6a6531b0a6, disabling compactions & flushes 2023-02-13 17:00:51,244 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:51,244 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:51,244 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. after waiting 0 ms 2023-02-13 17:00:51,245 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:51,245 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:51,245 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for 61523f42895e23c1110bbb6a6531b0a6: Waiting for close lock at 1676307651244Disabling compacts and flushes for region at 1676307651244Disabling writes for close at 1676307651245 (+1 ms)Writing region close event to WAL at 1676307651245Closed at 1676307651245 2023-02-13 17:00:51,245 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:51,245 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 52130edd005d4ce89422019dd11d135a from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,249 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:51,249 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 5f02a25012d1463cb5f37ea13b2cf2dd from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,249 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:51,249 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 15eaf8061d754363820c5e36fdb4956a from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,254 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d4a69b191ab64507915a551f7d3cf0ec from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,255 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 97a455c0271c489d81c6c7b49b2cf4d9 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,255 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 46cd66f26a1c432184d8807fe83439c0 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,256 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:51,256 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 47c584031cad45b68b3da5b7c3330a2d from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,257 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing 7925c8f572739336270bf824c12623ea, disabling compactions & flushes 2023-02-13 17:00:51,257 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:51,257 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:51,257 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. after waiting 0 ms 2023-02-13 17:00:51,257 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:51,257 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:51,257 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for 7925c8f572739336270bf824c12623ea: Waiting for close lock at 1676307651257Disabling compacts and flushes for region at 1676307651257Disabling writes for close at 1676307651257Writing region close event to WAL at 1676307651257Closed at 1676307651257 2023-02-13 17:00:51,258 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing bb0e3d66ca9f4f9ce3e87dc8bfc74c30, disabling compactions & flushes 2023-02-13 17:00:51,258 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:51,258 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:51,258 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing 2457c5d248f99f39cff76beaa71b4345, disabling compactions & flushes 2023-02-13 17:00:51,258 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. after waiting 0 ms 2023-02-13 17:00:51,258 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:51,258 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:51,258 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:51,258 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:51,258 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. after waiting 0 ms 2023-02-13 17:00:51,258 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Waiting for close lock at 1676307651258Disabling compacts and flushes for region at 1676307651258Disabling writes for close at 1676307651258Writing region close event to WAL at 1676307651258Closed at 1676307651258 2023-02-13 17:00:51,258 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:51,258 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:51,258 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for 2457c5d248f99f39cff76beaa71b4345: Waiting for close lock at 1676307651258Disabling compacts and flushes for region at 1676307651258Disabling writes for close at 1676307651258Writing region close event to WAL at 1676307651258Closed at 1676307651258 2023-02-13 17:00:51,260 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f6d566fa52ad45b6ba62c7872adbae6b from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,262 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing 0e16e4423db3157675739f103f8f5a30, disabling compactions & flushes 2023-02-13 17:00:51,262 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:51,262 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:51,262 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. after waiting 0 ms 2023-02-13 17:00:51,262 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:51,262 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:51,262 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for 0e16e4423db3157675739f103f8f5a30: Waiting for close lock at 1676307651262Disabling compacts and flushes for region at 1676307651262Disabling writes for close at 1676307651262Writing region close event to WAL at 1676307651262Closed at 1676307651262 2023-02-13 17:00:51,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 17:00:51,650 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:51,650 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 1b81c71679994dda95fb56406c0c1e6d from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,653 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 7909cb0a0160488495640804774da893 from cloned region in snapshot snaptb1-1676307630447 to table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:51,655 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing f31ed543f552e93833ffd6a17e17334b, disabling compactions & flushes 2023-02-13 17:00:51,655 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:51,655 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:51,655 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. after waiting 0 ms 2023-02-13 17:00:51,655 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:51,655 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:51,655 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for f31ed543f552e93833ffd6a17e17334b: Waiting for close lock at 1676307651655Disabling compacts and flushes for region at 1676307651655Disabling writes for close at 1676307651655Writing region close event to WAL at 1676307651655Closed at 1676307651655 2023-02-13 17:00:51,655 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307641126 type: DISABLED version: 2 ttl: 0 2023-02-13 17:00:51,655 INFO [PEWorker-5] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676307630447 on table=testRestoreSnapshot-1676307630447 completed! 2023-02-13 17:00:51,660 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307651660"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307651660"}]},"ts":"1676307651660"} 2023-02-13 17:00:51,660 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307651660"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307651660"}]},"ts":"1676307651660"} 2023-02-13 17:00:51,660 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307651660"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307651660"}]},"ts":"1676307651660"} 2023-02-13 17:00:51,660 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307651660"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307651660"}]},"ts":"1676307651660"} 2023-02-13 17:00:51,660 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307651660"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307651660"}]},"ts":"1676307651660"} 2023-02-13 17:00:51,660 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307651660"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307651660"}]},"ts":"1676307651660"} 2023-02-13 17:00:51,662 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 17:00:51,663 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307651663"}]},"ts":"1676307651663"} 2023-02-13 17:00:51,664 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLING in hbase:meta 2023-02-13 17:00:51,669 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:00:51,669 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:00:51,669 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:00:51,669 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:00:51,669 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:00:51,669 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN}] 2023-02-13 17:00:51,671 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN 2023-02-13 17:00:51,671 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN 2023-02-13 17:00:51,672 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN 2023-02-13 17:00:51,672 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN 2023-02-13 17:00:51,672 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=false 2023-02-13 17:00:51,672 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=false 2023-02-13 17:00:51,673 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=false 2023-02-13 17:00:51,673 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN 2023-02-13 17:00:51,673 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=false 2023-02-13 17:00:51,673 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN 2023-02-13 17:00:51,674 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=false 2023-02-13 17:00:51,674 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=false 2023-02-13 17:00:51,823 INFO [jenkins-hbase4:40255] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 17:00:51,826 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:51,826 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:51,826 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307651826"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307651826"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307651826"}]},"ts":"1676307651826"} 2023-02-13 17:00:51,826 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:51,826 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:51,826 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:51,826 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307651826"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307651826"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307651826"}]},"ts":"1676307651826"} 2023-02-13 17:00:51,826 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307651826"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307651826"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307651826"}]},"ts":"1676307651826"} 2023-02-13 17:00:51,826 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307651826"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307651826"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307651826"}]},"ts":"1676307651826"} 2023-02-13 17:00:51,826 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307651826"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307651826"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307651826"}]},"ts":"1676307651826"} 2023-02-13 17:00:51,827 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=179, state=RUNNABLE; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:51,830 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=180, state=RUNNABLE; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:51,831 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=176, state=RUNNABLE; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:51,832 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=178, state=RUNNABLE; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:51,833 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=181, state=RUNNABLE; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:51,833 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:51,833 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307651833"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307651833"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307651833"}]},"ts":"1676307651833"} 2023-02-13 17:00:51,835 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=177, state=RUNNABLE; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:51,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 17:00:51,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 17:00:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:51,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:51,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 17:00:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 17:00:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:51,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:51,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:51,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:51,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:51,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:51,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:51,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:51,986 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:51,987 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:51,987 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:51,987 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:51,987 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf 2023-02-13 17:00:51,987 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:51,988 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf 2023-02-13 17:00:51,987 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:51,988 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf 2023-02-13 17:00:51,988 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7925c8f572739336270bf824c12623ea columnFamilyName cf 2023-02-13 17:00:51,988 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bb0e3d66ca9f4f9ce3e87dc8bfc74c30 columnFamilyName cf 2023-02-13 17:00:51,988 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f31ed543f552e93833ffd6a17e17334b columnFamilyName cf 2023-02-13 17:00:51,996 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-5f02a25012d1463cb5f37ea13b2cf2dd 2023-02-13 17:00:51,997 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:51,997 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:52,003 DEBUG [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:52,003 INFO [StoreOpener-bb0e3d66ca9f4f9ce3e87dc8bfc74c30-1] regionserver.HStore(310): Store=bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:52,003 DEBUG [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-7909cb0a0160488495640804774da893 2023-02-13 17:00:52,004 INFO [StoreOpener-f31ed543f552e93833ffd6a17e17334b-1] regionserver.HStore(310): Store=f31ed543f552e93833ffd6a17e17334b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:52,004 DEBUG [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-d4a69b191ab64507915a551f7d3cf0ec 2023-02-13 17:00:52,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:52,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:52,005 INFO [StoreOpener-7925c8f572739336270bf824c12623ea-1] regionserver.HStore(310): Store=7925c8f572739336270bf824c12623ea/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:52,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:52,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:52,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:52,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:52,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:52,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:52,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:52,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 17:00:52,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 17:00:52,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f31ed543f552e93833ffd6a17e17334b; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59669772, jitterRate=-0.11085110902786255}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:52,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f31ed543f552e93833ffd6a17e17334b: Running coprocessor pre-open hook at 1676307651986Writing region info on filesystem at 1676307651986Initializing all the Stores at 1676307651986Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307651986Cleaning up temporary data from old regions at 1676307652005 (+19 ms)Cleaning up detritus from prior splits at 1676307652006 (+1 ms)Running coprocessor post-open hooks at 1676307652010 (+4 ms)Region opened successfully at 1676307652010 2023-02-13 17:00:52,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bb0e3d66ca9f4f9ce3e87dc8bfc74c30; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74210999, jitterRate=0.10583005845546722}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:52,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Running coprocessor pre-open hook at 1676307651986Writing region info on filesystem at 1676307651986Initializing all the Stores at 1676307651986Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307651986Cleaning up temporary data from old regions at 1676307652005 (+19 ms)Cleaning up detritus from prior splits at 1676307652006 (+1 ms)Running coprocessor post-open hooks at 1676307652010 (+4 ms)Region opened successfully at 1676307652010 2023-02-13 17:00:52,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b., pid=182, masterSystemTime=1676307651982 2023-02-13 17:00:52,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30., pid=185, masterSystemTime=1676307651983 2023-02-13 17:00:52,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 17:00:52,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7925c8f572739336270bf824c12623ea; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68476349, jitterRate=0.020377114415168762}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:52,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7925c8f572739336270bf824c12623ea: Running coprocessor pre-open hook at 1676307651985Writing region info on filesystem at 1676307651986 (+1 ms)Initializing all the Stores at 1676307651986Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307651986Cleaning up temporary data from old regions at 1676307652006 (+20 ms)Cleaning up detritus from prior splits at 1676307652006Running coprocessor post-open hooks at 1676307652011 (+5 ms)Region opened successfully at 1676307652011 2023-02-13 17:00:52,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:52,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:52,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:52,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 17:00:52,012 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:52,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:52,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:52,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:52,012 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307652012"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307652012"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307652012"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307652012"}]},"ts":"1676307652012"} 2023-02-13 17:00:52,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:52,013 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea., pid=183, masterSystemTime=1676307651982 2023-02-13 17:00:52,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:52,013 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:52,013 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:52,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:52,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 17:00:52,013 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307652013"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307652013"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307652013"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307652013"}]},"ts":"1676307652013"} 2023-02-13 17:00:52,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:52,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:52,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:52,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:52,014 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:52,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:52,014 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:52,014 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:52,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 17:00:52,015 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:52,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676307630447 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:52,015 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307652015"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307652015"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307652015"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307652015"}]},"ts":"1676307652015"} 2023-02-13 17:00:52,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:52,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:52,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:52,015 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:52,015 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf 2023-02-13 17:00:52,015 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:52,016 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2457c5d248f99f39cff76beaa71b4345 columnFamilyName cf 2023-02-13 17:00:52,016 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:52,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=179 2023-02-13 17:00:52,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=179, state=SUCCESS; OpenRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747 in 187 msec 2023-02-13 17:00:52,016 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:52,017 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf 2023-02-13 17:00:52,017 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=178 2023-02-13 17:00:52,017 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=178, state=SUCCESS; OpenRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 183 msec 2023-02-13 17:00:52,017 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:52,017 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf 2023-02-13 17:00:52,017 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e16e4423db3157675739f103f8f5a30 columnFamilyName cf 2023-02-13 17:00:52,017 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61523f42895e23c1110bbb6a6531b0a6 columnFamilyName cf 2023-02-13 17:00:52,017 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, ASSIGN in 347 msec 2023-02-13 17:00:52,018 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, ASSIGN in 348 msec 2023-02-13 17:00:52,018 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=180 2023-02-13 17:00:52,018 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=180, state=SUCCESS; OpenRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670 in 188 msec 2023-02-13 17:00:52,020 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, ASSIGN in 349 msec 2023-02-13 17:00:52,026 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:52,028 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:52,030 DEBUG [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-46cd66f26a1c432184d8807fe83439c0 2023-02-13 17:00:52,034 INFO [StoreOpener-2457c5d248f99f39cff76beaa71b4345-1] regionserver.HStore(310): Store=2457c5d248f99f39cff76beaa71b4345/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:52,035 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:52,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:52,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:52,036 DEBUG [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-f6d566fa52ad45b6ba62c7872adbae6b 2023-02-13 17:00:52,036 INFO [StoreOpener-0e16e4423db3157675739f103f8f5a30-1] regionserver.HStore(310): Store=0e16e4423db3157675739f103f8f5a30/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:52,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:52,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:52,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:52,039 DEBUG [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-f6bae3d5c579413fb7516ba3d60e58f8 2023-02-13 17:00:52,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 17:00:52,039 INFO [StoreOpener-61523f42895e23c1110bbb6a6531b0a6-1] regionserver.HStore(310): Store=61523f42895e23c1110bbb6a6531b0a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:52,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:52,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2457c5d248f99f39cff76beaa71b4345; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64329102, jitterRate=-0.04142168164253235}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:52,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2457c5d248f99f39cff76beaa71b4345: Running coprocessor pre-open hook at 1676307652013Writing region info on filesystem at 1676307652013Initializing all the Stores at 1676307652014 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307652014Cleaning up temporary data from old regions at 1676307652036 (+22 ms)Cleaning up detritus from prior splits at 1676307652036Running coprocessor post-open hooks at 1676307652040 (+4 ms)Region opened successfully at 1676307652040 2023-02-13 17:00:52,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:52,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345., pid=187, masterSystemTime=1676307651982 2023-02-13 17:00:52,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:52,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 17:00:52,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:52,042 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:52,042 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e16e4423db3157675739f103f8f5a30; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61620939, jitterRate=-0.0817764550447464}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:52,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e16e4423db3157675739f103f8f5a30: Running coprocessor pre-open hook at 1676307652013Writing region info on filesystem at 1676307652013Initializing all the Stores at 1676307652014 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307652014Cleaning up temporary data from old regions at 1676307652038 (+24 ms)Cleaning up detritus from prior splits at 1676307652038Running coprocessor post-open hooks at 1676307652042 (+4 ms)Region opened successfully at 1676307652042 2023-02-13 17:00:52,043 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:52,043 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307652043"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307652043"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307652043"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307652043"}]},"ts":"1676307652043"} 2023-02-13 17:00:52,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30., pid=184, masterSystemTime=1676307651983 2023-02-13 17:00:52,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:52,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:52,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:52,045 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:52,045 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307652045"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307652045"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307652045"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307652045"}]},"ts":"1676307652045"} 2023-02-13 17:00:52,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 17:00:52,046 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 61523f42895e23c1110bbb6a6531b0a6; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68830452, jitterRate=0.0256536602973938}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:52,046 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=177 2023-02-13 17:00:52,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 61523f42895e23c1110bbb6a6531b0a6: Running coprocessor pre-open hook at 1676307652015Writing region info on filesystem at 1676307652015Initializing all the Stores at 1676307652016 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307652016Cleaning up temporary data from old regions at 1676307652041 (+25 ms)Cleaning up detritus from prior splits at 1676307652041Running coprocessor post-open hooks at 1676307652046 (+5 ms)Region opened successfully at 1676307652046 2023-02-13 17:00:52,046 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=177, state=SUCCESS; OpenRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,34035,1676307627747 in 210 msec 2023-02-13 17:00:52,047 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, ASSIGN in 377 msec 2023-02-13 17:00:52,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6., pid=186, masterSystemTime=1676307651982 2023-02-13 17:00:52,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:52,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:52,049 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:52,049 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307652049"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307652049"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307652049"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307652049"}]},"ts":"1676307652049"} 2023-02-13 17:00:52,050 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=176 2023-02-13 17:00:52,050 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=176, state=SUCCESS; OpenRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 217 msec 2023-02-13 17:00:52,051 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, ASSIGN in 381 msec 2023-02-13 17:00:52,052 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=181 2023-02-13 17:00:52,052 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=181, state=SUCCESS; OpenRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670 in 218 msec 2023-02-13 17:00:52,053 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=181, resume processing ppid=175 2023-02-13 17:00:52,053 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, ASSIGN in 383 msec 2023-02-13 17:00:52,054 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307652054"}]},"ts":"1676307652054"} 2023-02-13 17:00:52,055 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=ENABLED in hbase:meta 2023-02-13 17:00:52,059 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676307630447 snapshot=name: "snaptb1-1676307630447" table: "testRestoreSnapshot-1676307630447" creation_time: 1676307641126 type: DISABLED version: 2 ttl: 0 ) in 889 msec 2023-02-13 17:00:52,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 17:00:52,680 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676307630447, procId: 175 completed 2023-02-13 17:00:52,681 DEBUG [Listener at localhost/40167] zookeeper.ReadOnlyZKClient(139): Connect 0x594888d9 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:52,689 DEBUG [Listener at localhost/40167] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7254d96e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:52,691 DEBUG [hconnection-0x1913e027-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:52,692 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55280, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:52,695 DEBUG [hconnection-0x1913e027-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:52,696 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39412, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:52,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,893 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 17:00:52,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:52,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,093 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 17:00:53,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,241 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 17:00:53,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,378 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 17:00:53,381 DEBUG [hconnection-0x1913e027-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:53,382 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32970, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:53,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,565 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 17:00:53,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:53,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,373 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x131d42f3 to 127.0.0.1:54083 2023-02-13 17:00:54,373 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:54,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:54,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:00:55,400 INFO [Listener at localhost/40167] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676307630447 2023-02-13 17:00:55,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676307630447 2023-02-13 17:00:55,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:55,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 17:00:55,405 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307655405"}]},"ts":"1676307655405"} 2023-02-13 17:00:55,406 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLING in hbase:meta 2023-02-13 17:00:55,408 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676307630447 to state=DISABLING 2023-02-13 17:00:55,409 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN}] 2023-02-13 17:00:55,411 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN 2023-02-13 17:00:55,411 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN 2023-02-13 17:00:55,411 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN 2023-02-13 17:00:55,411 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN 2023-02-13 17:00:55,411 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN 2023-02-13 17:00:55,412 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:55,412 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:55,412 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:55,412 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307655412"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307655412"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307655412"}]},"ts":"1676307655412"} 2023-02-13 17:00:55,413 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307655412"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307655412"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307655412"}]},"ts":"1676307655412"} 2023-02-13 17:00:55,412 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:55,413 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307655412"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307655412"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307655412"}]},"ts":"1676307655412"} 2023-02-13 17:00:55,412 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:55,412 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307655412"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307655412"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307655412"}]},"ts":"1676307655412"} 2023-02-13 17:00:55,413 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307655412"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307655412"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307655412"}]},"ts":"1676307655412"} 2023-02-13 17:00:55,414 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=191, state=RUNNABLE; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:55,415 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=194, state=RUNNABLE; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:55,416 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=192, state=RUNNABLE; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:55,419 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=189, state=RUNNABLE; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:55,420 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN 2023-02-13 17:00:55,420 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=193, state=RUNNABLE; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:55,421 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:55,421 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307655420"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307655420"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307655420"}]},"ts":"1676307655420"} 2023-02-13 17:00:55,422 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=190, state=RUNNABLE; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:55,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:55,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e16e4423db3157675739f103f8f5a30, disabling compactions & flushes 2023-02-13 17:00:55,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:55,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:55,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. after waiting 0 ms 2023-02-13 17:00:55,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:55,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:55,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7925c8f572739336270bf824c12623ea, disabling compactions & flushes 2023-02-13 17:00:55,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:55,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:55,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. after waiting 0 ms 2023-02-13 17:00:55,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:55,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:55,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f31ed543f552e93833ffd6a17e17334b, disabling compactions & flushes 2023-02-13 17:00:55,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:55,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:55,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. after waiting 0 ms 2023-02-13 17:00:55,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:55,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 17:00:55,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 17:00:55,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30. 2023-02-13 17:00:55,583 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e16e4423db3157675739f103f8f5a30: Waiting for close lock at 1676307655567Running coprocessor pre-close hooks at 1676307655567Disabling compacts and flushes for region at 1676307655567Disabling writes for close at 1676307655568 (+1 ms)Writing region close event to WAL at 1676307655572 (+4 ms)Running coprocessor post-close hooks at 1676307655583 (+11 ms)Closed at 1676307655583 2023-02-13 17:00:55,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea. 2023-02-13 17:00:55,583 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 17:00:55,583 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7925c8f572739336270bf824c12623ea: Waiting for close lock at 1676307655568Running coprocessor pre-close hooks at 1676307655568Disabling compacts and flushes for region at 1676307655568Disabling writes for close at 1676307655572 (+4 ms)Writing region close event to WAL at 1676307655576 (+4 ms)Running coprocessor post-close hooks at 1676307655583 (+7 ms)Closed at 1676307655583 2023-02-13 17:00:55,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b. 2023-02-13 17:00:55,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f31ed543f552e93833ffd6a17e17334b: Waiting for close lock at 1676307655572Running coprocessor pre-close hooks at 1676307655572Disabling compacts and flushes for region at 1676307655572Disabling writes for close at 1676307655576 (+4 ms)Writing region close event to WAL at 1676307655579 (+3 ms)Running coprocessor post-close hooks at 1676307655584 (+5 ms)Closed at 1676307655584 2023-02-13 17:00:55,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:55,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:55,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bb0e3d66ca9f4f9ce3e87dc8bfc74c30, disabling compactions & flushes 2023-02-13 17:00:55,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:55,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:55,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. after waiting 0 ms 2023-02-13 17:00:55,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:55,586 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=0e16e4423db3157675739f103f8f5a30, regionState=CLOSED 2023-02-13 17:00:55,586 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307655586"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307655586"}]},"ts":"1676307655586"} 2023-02-13 17:00:55,586 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7925c8f572739336270bf824c12623ea 2023-02-13 17:00:55,586 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:55,587 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 61523f42895e23c1110bbb6a6531b0a6, disabling compactions & flushes 2023-02-13 17:00:55,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:55,587 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:55,587 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. after waiting 0 ms 2023-02-13 17:00:55,587 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:55,587 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=7925c8f572739336270bf824c12623ea, regionState=CLOSED 2023-02-13 17:00:55,594 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307655587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307655587"}]},"ts":"1676307655587"} 2023-02-13 17:00:55,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:55,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:55,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2457c5d248f99f39cff76beaa71b4345, disabling compactions & flushes 2023-02-13 17:00:55,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:55,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:55,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. after waiting 0 ms 2023-02-13 17:00:55,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:55,596 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=f31ed543f552e93833ffd6a17e17334b, regionState=CLOSED 2023-02-13 17:00:55,596 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307655596"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307655596"}]},"ts":"1676307655596"} 2023-02-13 17:00:55,598 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=189 2023-02-13 17:00:55,598 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=189, state=SUCCESS; CloseRegionProcedure 0e16e4423db3157675739f103f8f5a30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 179 msec 2023-02-13 17:00:55,599 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=193 2023-02-13 17:00:55,599 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=193, state=SUCCESS; CloseRegionProcedure 7925c8f572739336270bf824c12623ea, server=jenkins-hbase4.apache.org,43327,1676307627670 in 177 msec 2023-02-13 17:00:55,601 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=192 2023-02-13 17:00:55,601 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=7925c8f572739336270bf824c12623ea, UNASSIGN in 190 msec 2023-02-13 17:00:55,601 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=192, state=SUCCESS; CloseRegionProcedure f31ed543f552e93833ffd6a17e17334b, server=jenkins-hbase4.apache.org,34035,1676307627747 in 182 msec 2023-02-13 17:00:55,601 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=0e16e4423db3157675739f103f8f5a30, UNASSIGN in 189 msec 2023-02-13 17:00:55,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 17:00:55,603 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=f31ed543f552e93833ffd6a17e17334b, UNASSIGN in 192 msec 2023-02-13 17:00:55,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6. 2023-02-13 17:00:55,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 61523f42895e23c1110bbb6a6531b0a6: Waiting for close lock at 1676307655586Running coprocessor pre-close hooks at 1676307655586Disabling compacts and flushes for region at 1676307655586Disabling writes for close at 1676307655587 (+1 ms)Writing region close event to WAL at 1676307655595 (+8 ms)Running coprocessor post-close hooks at 1676307655603 (+8 ms)Closed at 1676307655603 2023-02-13 17:00:55,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 17:00:55,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345. 2023-02-13 17:00:55,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2457c5d248f99f39cff76beaa71b4345: Waiting for close lock at 1676307655594Running coprocessor pre-close hooks at 1676307655594Disabling compacts and flushes for region at 1676307655594Disabling writes for close at 1676307655595 (+1 ms)Writing region close event to WAL at 1676307655599 (+4 ms)Running coprocessor post-close hooks at 1676307655604 (+5 ms)Closed at 1676307655604 2023-02-13 17:00:55,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:55,605 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=61523f42895e23c1110bbb6a6531b0a6, regionState=CLOSED 2023-02-13 17:00:55,605 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676307655604"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307655604"}]},"ts":"1676307655604"} 2023-02-13 17:00:55,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:55,606 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=2457c5d248f99f39cff76beaa71b4345, regionState=CLOSED 2023-02-13 17:00:55,606 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307655606"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307655606"}]},"ts":"1676307655606"} 2023-02-13 17:00:55,607 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=194 2023-02-13 17:00:55,607 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=194, state=SUCCESS; CloseRegionProcedure 61523f42895e23c1110bbb6a6531b0a6, server=jenkins-hbase4.apache.org,43327,1676307627670 in 191 msec 2023-02-13 17:00:55,609 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=61523f42895e23c1110bbb6a6531b0a6, UNASSIGN in 198 msec 2023-02-13 17:00:55,609 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=190 2023-02-13 17:00:55,609 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=190, state=SUCCESS; CloseRegionProcedure 2457c5d248f99f39cff76beaa71b4345, server=jenkins-hbase4.apache.org,34035,1676307627747 in 185 msec 2023-02-13 17:00:55,609 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=2457c5d248f99f39cff76beaa71b4345, UNASSIGN in 200 msec 2023-02-13 17:00:55,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 17:00:55,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30. 2023-02-13 17:00:55,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bb0e3d66ca9f4f9ce3e87dc8bfc74c30: Waiting for close lock at 1676307655585Running coprocessor pre-close hooks at 1676307655585Disabling compacts and flushes for region at 1676307655585Disabling writes for close at 1676307655585Writing region close event to WAL at 1676307655595 (+10 ms)Running coprocessor post-close hooks at 1676307655612 (+17 ms)Closed at 1676307655612 2023-02-13 17:00:55,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:55,613 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, regionState=CLOSED 2023-02-13 17:00:55,614 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676307655613"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307655613"}]},"ts":"1676307655613"} 2023-02-13 17:00:55,616 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=191 2023-02-13 17:00:55,616 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=191, state=SUCCESS; CloseRegionProcedure bb0e3d66ca9f4f9ce3e87dc8bfc74c30, server=jenkins-hbase4.apache.org,43065,1676307627716 in 201 msec 2023-02-13 17:00:55,617 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=191, resume processing ppid=188 2023-02-13 17:00:55,617 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676307630447, region=bb0e3d66ca9f4f9ce3e87dc8bfc74c30, UNASSIGN in 207 msec 2023-02-13 17:00:55,617 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307655617"}]},"ts":"1676307655617"} 2023-02-13 17:00:55,618 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676307630447, state=DISABLED in hbase:meta 2023-02-13 17:00:55,621 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676307630447 to state=DISABLED 2023-02-13 17:00:55,623 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676307630447 in 222 msec 2023-02-13 17:00:55,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 17:00:55,656 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 188 completed 2023-02-13 17:00:55,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676307630447 2023-02-13 17:00:55,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:55,659 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:55,659 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:55,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-13 17:00:55,666 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:55,666 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:55,666 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:55,666 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:55,666 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:55,666 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:55,667 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits] 2023-02-13 17:00:55,667 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits] 2023-02-13 17:00:55,668 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits] 2023-02-13 17:00:55,668 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits] 2023-02-13 17:00:55,668 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits] 2023-02-13 17:00:55,668 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits] 2023-02-13 17:00:55,671 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-52130edd005d4ce89422019dd11d135a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-52130edd005d4ce89422019dd11d135a. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650907 currentMtime: 1676307651254 2023-02-13 17:00:55,671 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-1b81c71679994dda95fb56406c0c1e6d already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-1b81c71679994dda95fb56406c0c1e6d. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650907 currentMtime: 1676307651652 2023-02-13 17:00:55,671 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-47c584031cad45b68b3da5b7c3330a2d already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-47c584031cad45b68b3da5b7c3330a2d. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650907 currentMtime: 1676307651260 2023-02-13 17:00:55,672 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-5f02a25012d1463cb5f37ea13b2cf2dd already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-5f02a25012d1463cb5f37ea13b2cf2dd. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650908 currentMtime: 1676307651255 2023-02-13 17:00:55,672 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-52130edd005d4ce89422019dd11d135a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-52130edd005d4ce89422019dd11d135a.1676307655667. 2023-02-13 17:00:55,672 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-1b81c71679994dda95fb56406c0c1e6d to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-1b81c71679994dda95fb56406c0c1e6d.1676307655668. 2023-02-13 17:00:55,672 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-522b6eff1a16428ab53f7cfcc13f5481 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-522b6eff1a16428ab53f7cfcc13f5481. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650906 currentMtime: 1676307651239 2023-02-13 17:00:55,672 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-47c584031cad45b68b3da5b7c3330a2d to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-47c584031cad45b68b3da5b7c3330a2d.1676307655668. 2023-02-13 17:00:55,673 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-5f02a25012d1463cb5f37ea13b2cf2dd to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-5f02a25012d1463cb5f37ea13b2cf2dd.1676307655667. 2023-02-13 17:00:55,673 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-522b6eff1a16428ab53f7cfcc13f5481 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-522b6eff1a16428ab53f7cfcc13f5481.1676307655668. 2023-02-13 17:00:55,676 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-1b81c71679994dda95fb56406c0c1e6d to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-1b81c71679994dda95fb56406c0c1e6d 2023-02-13 17:00:55,676 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-15eaf8061d754363820c5e36fdb4956a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-15eaf8061d754363820c5e36fdb4956a. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650907 currentMtime: 1676307651255 2023-02-13 17:00:55,676 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-52130edd005d4ce89422019dd11d135a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-52130edd005d4ce89422019dd11d135a 2023-02-13 17:00:55,677 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-15eaf8061d754363820c5e36fdb4956a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-15eaf8061d754363820c5e36fdb4956a.1676307655668. 2023-02-13 17:00:55,677 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-47c584031cad45b68b3da5b7c3330a2d to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-47c584031cad45b68b3da5b7c3330a2d 2023-02-13 17:00:55,677 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-522b6eff1a16428ab53f7cfcc13f5481 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-522b6eff1a16428ab53f7cfcc13f5481 2023-02-13 17:00:55,677 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-5f02a25012d1463cb5f37ea13b2cf2dd to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-5f02a25012d1463cb5f37ea13b2cf2dd 2023-02-13 17:00:55,678 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-15eaf8061d754363820c5e36fdb4956a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-15eaf8061d754363820c5e36fdb4956a 2023-02-13 17:00:55,678 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-7909cb0a0160488495640804774da893 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-7909cb0a0160488495640804774da893. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650909 currentMtime: 1676307651654 2023-02-13 17:00:55,678 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-7909cb0a0160488495640804774da893 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-7909cb0a0160488495640804774da893.1676307655668. 2023-02-13 17:00:55,679 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-d4a69b191ab64507915a551f7d3cf0ec already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-d4a69b191ab64507915a551f7d3cf0ec. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650909 currentMtime: 1676307651257 2023-02-13 17:00:55,679 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-97a455c0271c489d81c6c7b49b2cf4d9 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-97a455c0271c489d81c6c7b49b2cf4d9. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650910 currentMtime: 1676307651258 2023-02-13 17:00:55,679 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-f6bae3d5c579413fb7516ba3d60e58f8 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-f6bae3d5c579413fb7516ba3d60e58f8. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650909 currentMtime: 1676307651242 2023-02-13 17:00:55,679 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-f6d566fa52ad45b6ba62c7872adbae6b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-f6d566fa52ad45b6ba62c7872adbae6b. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650910 currentMtime: 1676307651262 2023-02-13 17:00:55,679 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-d4a69b191ab64507915a551f7d3cf0ec to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-d4a69b191ab64507915a551f7d3cf0ec.1676307655667. 2023-02-13 17:00:55,679 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-97a455c0271c489d81c6c7b49b2cf4d9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-97a455c0271c489d81c6c7b49b2cf4d9.1676307655667. 2023-02-13 17:00:55,679 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-f6bae3d5c579413fb7516ba3d60e58f8 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-f6bae3d5c579413fb7516ba3d60e58f8.1676307655668. 2023-02-13 17:00:55,680 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-f6d566fa52ad45b6ba62c7872adbae6b to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-f6d566fa52ad45b6ba62c7872adbae6b.1676307655668. 2023-02-13 17:00:55,680 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-46cd66f26a1c432184d8807fe83439c0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-46cd66f26a1c432184d8807fe83439c0. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650909 currentMtime: 1676307651258 2023-02-13 17:00:55,680 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-7909cb0a0160488495640804774da893 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/cf/testRestoreSnapshot-1676307630447=f31ed543f552e93833ffd6a17e17334b-7909cb0a0160488495640804774da893 2023-02-13 17:00:55,680 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-d4a69b191ab64507915a551f7d3cf0ec to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/cf/testRestoreSnapshot-1676307630447=7925c8f572739336270bf824c12623ea-d4a69b191ab64507915a551f7d3cf0ec 2023-02-13 17:00:55,681 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-97a455c0271c489d81c6c7b49b2cf4d9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/cf/testRestoreSnapshot-1676307630447=bb0e3d66ca9f4f9ce3e87dc8bfc74c30-97a455c0271c489d81c6c7b49b2cf4d9 2023-02-13 17:00:55,681 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-46cd66f26a1c432184d8807fe83439c0 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-46cd66f26a1c432184d8807fe83439c0.1676307655668. 2023-02-13 17:00:55,681 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-f6bae3d5c579413fb7516ba3d60e58f8 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/cf/testRestoreSnapshot-1676307630447=61523f42895e23c1110bbb6a6531b0a6-f6bae3d5c579413fb7516ba3d60e58f8 2023-02-13 17:00:55,681 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-f6d566fa52ad45b6ba62c7872adbae6b to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/cf/testRestoreSnapshot-1676307630447=0e16e4423db3157675739f103f8f5a30-f6d566fa52ad45b6ba62c7872adbae6b 2023-02-13 17:00:55,682 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-46cd66f26a1c432184d8807fe83439c0 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/cf/testRestoreSnapshot-1676307630447=2457c5d248f99f39cff76beaa71b4345-46cd66f26a1c432184d8807fe83439c0 2023-02-13 17:00:55,684 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30/recovered.edits/18.seqid 2023-02-13 17:00:55,684 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6/recovered.edits/18.seqid 2023-02-13 17:00:55,684 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea/recovered.edits/18.seqid 2023-02-13 17:00:55,685 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30/recovered.edits/18.seqid 2023-02-13 17:00:55,685 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/bb0e3d66ca9f4f9ce3e87dc8bfc74c30 2023-02-13 17:00:55,685 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b/recovered.edits/18.seqid 2023-02-13 17:00:55,685 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/7925c8f572739336270bf824c12623ea 2023-02-13 17:00:55,685 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/61523f42895e23c1110bbb6a6531b0a6 2023-02-13 17:00:55,685 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345/recovered.edits/18.seqid 2023-02-13 17:00:55,685 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/f31ed543f552e93833ffd6a17e17334b 2023-02-13 17:00:55,685 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/0e16e4423db3157675739f103f8f5a30 2023-02-13 17:00:55,686 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testRestoreSnapshot-1676307630447/2457c5d248f99f39cff76beaa71b4345 2023-02-13 17:00:55,686 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676307630447 regions 2023-02-13 17:00:55,686 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8 2023-02-13 17:00:55,687 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf] 2023-02-13 17:00:55,688 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650918 currentMtime: 1676307651192 2023-02-13 17:00:55,689 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a.1676307655687. 2023-02-13 17:00:55,690 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c2023021326dd77d03a6846a4aaa0ee0b09c5511a 2023-02-13 17:00:55,691 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650919 currentMtime: 1676307651194 2023-02-13 17:00:55,691 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9.1676307655687. 2023-02-13 17:00:55,692 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-a87ff679a2f3e71d9181a67b7542122c20230213f572ea8d7fa7446e9838ee0e5fef92d9 2023-02-13 17:00:55,693 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650921 currentMtime: 1676307651196 2023-02-13 17:00:55,693 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9.1676307655687. 2023-02-13 17:00:55,694 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b2023021304c735e314f040c69950e8c05fa21db9 2023-02-13 17:00:55,694 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650922 currentMtime: 1676307651198 2023-02-13 17:00:55,695 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738.1676307655687. 2023-02-13 17:00:55,695 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c4ca4238a0b923820dcc509a6f75849b20230213bbf3769caf90425bbb61c167c92b1738 2023-02-13 17:00:55,696 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650923 currentMtime: 1676307651199 2023-02-13 17:00:55,697 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92.1676307655687. 2023-02-13 17:00:55,697 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213afef8a4528b042ae88c4e9363329da92 2023-02-13 17:00:55,698 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650924 currentMtime: 1676307651201 2023-02-13 17:00:55,698 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856.1676307655687. 2023-02-13 17:00:55,699 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-c81e728d9d4c2f636f067f89cc14862c20230213b085573754f54a058189a0666b421856 2023-02-13 17:00:55,700 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650925 currentMtime: 1676307651203 2023-02-13 17:00:55,700 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9.1676307655687. 2023-02-13 17:00:55,701 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302133fe1a754140e464e81eaa91f26b5a7e9 2023-02-13 17:00:55,702 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650926 currentMtime: 1676307651205 2023-02-13 17:00:55,702 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70.1676307655687. 2023-02-13 17:00:55,703 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-d41d8cd98f00b204e9800998ecf8427e202302138172d824d7134349938552ac8c581f70 2023-02-13 17:00:55,703 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650927 currentMtime: 1676307651206 2023-02-13 17:00:55,704 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198.1676307655687. 2023-02-13 17:00:55,704 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d52023021381fc97ac2076411892972701f5bee198 2023-02-13 17:00:55,705 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650928 currentMtime: 1676307651208 2023-02-13 17:00:55,705 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4.1676307655687. 2023-02-13 17:00:55,706 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-e4da3b7fbbce2345d7772b0674a318d520230213e3a931c7718f416ab7d7b4e12ac570e4 2023-02-13 17:00:55,707 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650929 currentMtime: 1676307651211 2023-02-13 17:00:55,707 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4.1676307655687. 2023-02-13 17:00:55,708 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132c522d494df1469397293f7a8e12a4c4 2023-02-13 17:00:55,709 ERROR [PEWorker-2] backup.HFileArchiver(523): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1. archiveLen: 0 currentLen: 0 archiveMtime: 1676307650930 currentMtime: 1676307651213 2023-02-13 17:00:55,709 INFO [PEWorker-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1.1676307655687. 2023-02-13 17:00:55,709 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8/cf/testRestoreSnapshot-1676307630447=7469c224c9ad52a75eb8915b71d139b8-eccbc87e4b5ce2fe28308fd9f2a7baf32023021386120e7c34cc4009a512658179c627e1 2023-02-13 17:00:55,710 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testRestoreSnapshot-1676307630447/7469c224c9ad52a75eb8915b71d139b8 2023-02-13 17:00:55,711 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:55,714 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676307630447 from hbase:meta 2023-02-13 17:00:55,716 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676307630447' descriptor. 2023-02-13 17:00:55,717 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:55,717 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676307630447' from region states. 2023-02-13 17:00:55,717 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307655717"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:55,717 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307655717"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:55,717 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307655717"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:55,717 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307655717"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:55,717 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307655717"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:55,717 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307655717"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:55,719 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 17:00:55,719 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0e16e4423db3157675739f103f8f5a30, NAME => 'testRestoreSnapshot-1676307630447,,1676307630470.0e16e4423db3157675739f103f8f5a30.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 2457c5d248f99f39cff76beaa71b4345, NAME => 'testRestoreSnapshot-1676307630447,1,1676307630470.2457c5d248f99f39cff76beaa71b4345.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => bb0e3d66ca9f4f9ce3e87dc8bfc74c30, NAME => 'testRestoreSnapshot-1676307630447,2,1676307630470.bb0e3d66ca9f4f9ce3e87dc8bfc74c30.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f31ed543f552e93833ffd6a17e17334b, NAME => 'testRestoreSnapshot-1676307630447,3,1676307630470.f31ed543f552e93833ffd6a17e17334b.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 7925c8f572739336270bf824c12623ea, NAME => 'testRestoreSnapshot-1676307630447,4,1676307630470.7925c8f572739336270bf824c12623ea.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 61523f42895e23c1110bbb6a6531b0a6, NAME => 'testRestoreSnapshot-1676307630447,5,1676307630470.61523f42895e23c1110bbb6a6531b0a6.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 17:00:55,719 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676307630447' as deleted. 2023-02-13 17:00:55,719 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676307630447","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676307655719"}]},"ts":"9223372036854775807"} 2023-02-13 17:00:55,720 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676307630447 state from META 2023-02-13 17:00:55,722 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 2023-02-13 17:00:55,723 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676307630447 in 65 msec 2023-02-13 17:00:55,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-13 17:00:55,911 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676307630447, procId: 201 completed 2023-02-13 17:00:55,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676307630447" 2023-02-13 17:00:55,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676307630447 2023-02-13 17:00:55,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676307630447" 2023-02-13 17:00:55,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676307630447 2023-02-13 17:00:55,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1676307630447" 2023-02-13 17:00:55,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676307630447 2023-02-13 17:00:55,950 INFO [Listener at localhost/40167] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=422 (was 357) Potentially hanging thread: hconnection-0x60dfe106-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1146786389_15 at /127.0.0.1:35410 [Waiting for operation #519] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1146786389_15 at /127.0.0.1:35510 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1146786389_15 at /127.0.0.1:51330 [Waiting for operation #761] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54083@0x594888d9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/755824594.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1022038589_15 at /127.0.0.1:46732 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1022038589_15 at /127.0.0.1:46720 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54083@0x594888d9-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x60dfe106-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x60dfe106-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7b9b1635-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1022038589_15 at /127.0.0.1:43630 [Waiting for operation #476] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-690826998_15 at /127.0.0.1:43530 [Waiting for operation #498] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-690826998_15 at /127.0.0.1:43618 [Waiting for operation #478] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1566478913_15 at /127.0.0.1:35504 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1022038589_15 at /127.0.0.1:46766 [Waiting for operation #526] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1022038589_15 at /127.0.0.1:46722 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x60dfe106-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-690826998_15 at /127.0.0.1:46754 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1658626342_15 at /127.0.0.1:35526 [Waiting for operation #527] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-690826998_15 at /127.0.0.1:35508 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1022038589_15 at /127.0.0.1:46744 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1566478913_15 at /127.0.0.1:35390 [Waiting for operation #522] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54083@0x594888d9-SendThread(127.0.0.1:54083) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) - Thread LEAK? -, OpenFileDescriptor=809 (was 693) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=299 (was 244) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=880 (was 1405) 2023-02-13 17:00:55,966 INFO [Listener at localhost/40167] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=422, OpenFileDescriptor=809, MaxFileDescriptor=60000, SystemLoadAverage=299, ProcessCount=168, AvailableMemoryMB=879 2023-02-13 17:00:55,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1676307655966', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-13 17:00:55,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:00:55,971 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676307655966 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:00:55,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676307655966" procId is: 202 2023-02-13 17:00:55,972 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676307655966 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:00:55,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 17:00:55,976 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:55,976 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:55,976 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:55,976 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:55,976 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:55,976 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:55,976 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 empty. 2023-02-13 17:00:55,976 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf empty. 2023-02-13 17:00:55,977 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 empty. 2023-02-13 17:00:55,977 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 empty. 2023-02-13 17:00:55,977 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 empty. 2023-02-13 17:00:55,977 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 empty. 2023-02-13 17:00:55,977 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:55,977 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:55,977 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:55,977 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:55,978 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:55,978 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:55,978 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676307655966 regions 2023-02-13 17:00:55,989 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/.tabledesc/.tableinfo.0000000001 2023-02-13 17:00:55,990 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(7671): creating {ENCODED => 8d87bc00cf39a8c35d32e1f7fb7f3db5, NAME => 'testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676307655966', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:55,991 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-1] regionserver.HRegion(7671): creating {ENCODED => d204882d59ec244668fd188df03d63c2, NAME => 'testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676307655966', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:55,991 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(7671): creating {ENCODED => f570113e41c274b8a0b2582dd682ff81, NAME => 'testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676307655966', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:56,010 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,010 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1603): Closing 8d87bc00cf39a8c35d32e1f7fb7f3db5, disabling compactions & flushes 2023-02-13 17:00:56,010 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:56,010 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:56,010 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. after waiting 0 ms 2023-02-13 17:00:56,010 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:56,010 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:56,011 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1557): Region close journal for 8d87bc00cf39a8c35d32e1f7fb7f3db5: Waiting for close lock at 1676307656010Disabling compacts and flushes for region at 1676307656010Disabling writes for close at 1676307656010Writing region close event to WAL at 1676307656010Closed at 1676307656010 2023-02-13 17:00:56,011 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(7671): creating {ENCODED => 14fbb78d336eb28a243b7c7310fa32d2, NAME => 'testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676307655966', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:56,015 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,016 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1603): Closing f570113e41c274b8a0b2582dd682ff81, disabling compactions & flushes 2023-02-13 17:00:56,016 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:56,016 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:56,016 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. after waiting 0 ms 2023-02-13 17:00:56,016 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:56,016 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:56,016 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1557): Region close journal for f570113e41c274b8a0b2582dd682ff81: Waiting for close lock at 1676307656016Disabling compacts and flushes for region at 1676307656016Disabling writes for close at 1676307656016Writing region close event to WAL at 1676307656016Closed at 1676307656016 2023-02-13 17:00:56,017 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(7671): creating {ENCODED => 14b9e8c5d6a0e94884bc0b364ef904bf, NAME => 'testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676307655966', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:56,020 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,020 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1603): Closing 14fbb78d336eb28a243b7c7310fa32d2, disabling compactions & flushes 2023-02-13 17:00:56,020 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:56,020 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:56,020 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. after waiting 0 ms 2023-02-13 17:00:56,020 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:56,020 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:56,020 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1557): Region close journal for 14fbb78d336eb28a243b7c7310fa32d2: Waiting for close lock at 1676307656020Disabling compacts and flushes for region at 1676307656020Disabling writes for close at 1676307656020Writing region close event to WAL at 1676307656020Closed at 1676307656020 2023-02-13 17:00:56,020 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(7671): creating {ENCODED => b1c0a10356cfd3754fa8161677e0dd55, NAME => 'testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676307655966', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp 2023-02-13 17:00:56,034 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,035 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1603): Closing b1c0a10356cfd3754fa8161677e0dd55, disabling compactions & flushes 2023-02-13 17:00:56,035 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:56,035 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:56,035 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. after waiting 0 ms 2023-02-13 17:00:56,035 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:56,035 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:56,035 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-2] regionserver.HRegion(1557): Region close journal for b1c0a10356cfd3754fa8161677e0dd55: Waiting for close lock at 1676307656035Disabling compacts and flushes for region at 1676307656035Disabling writes for close at 1676307656035Writing region close event to WAL at 1676307656035Closed at 1676307656035 2023-02-13 17:00:56,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 17:00:56,309 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 17:00:56,412 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,413 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-1] regionserver.HRegion(1603): Closing d204882d59ec244668fd188df03d63c2, disabling compactions & flushes 2023-02-13 17:00:56,413 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:56,413 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:56,413 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. after waiting 0 ms 2023-02-13 17:00:56,413 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:56,413 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:56,413 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-1] regionserver.HRegion(1557): Region close journal for d204882d59ec244668fd188df03d63c2: Waiting for close lock at 1676307656413Disabling compacts and flushes for region at 1676307656413Disabling writes for close at 1676307656413Writing region close event to WAL at 1676307656413Closed at 1676307656413 2023-02-13 17:00:56,429 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,429 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1603): Closing 14b9e8c5d6a0e94884bc0b364ef904bf, disabling compactions & flushes 2023-02-13 17:00:56,429 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:56,429 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:56,429 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. after waiting 0 ms 2023-02-13 17:00:56,429 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:56,429 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:56,429 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676307655966-pool-0] regionserver.HRegion(1557): Region close journal for 14b9e8c5d6a0e94884bc0b364ef904bf: Waiting for close lock at 1676307656429Disabling compacts and flushes for region at 1676307656429Disabling writes for close at 1676307656429Writing region close event to WAL at 1676307656429Closed at 1676307656429 2023-02-13 17:00:56,431 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676307655966 execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:00:56,432 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656432"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307656432"}]},"ts":"1676307656432"} 2023-02-13 17:00:56,432 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307656432"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307656432"}]},"ts":"1676307656432"} 2023-02-13 17:00:56,432 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656432"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307656432"}]},"ts":"1676307656432"} 2023-02-13 17:00:56,432 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307656432"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307656432"}]},"ts":"1676307656432"} 2023-02-13 17:00:56,432 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656432"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307656432"}]},"ts":"1676307656432"} 2023-02-13 17:00:56,432 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656432"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307656432"}]},"ts":"1676307656432"} 2023-02-13 17:00:56,436 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 17:00:56,437 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676307655966 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:00:56,437 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307656437"}]},"ts":"1676307656437"} 2023-02-13 17:00:56,438 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=ENABLING in hbase:meta 2023-02-13 17:00:56,442 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:00:56,445 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:00:56,445 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:00:56,445 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:00:56,445 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:00:56,445 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN}] 2023-02-13 17:00:56,447 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN 2023-02-13 17:00:56,447 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN 2023-02-13 17:00:56,447 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN 2023-02-13 17:00:56,447 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN 2023-02-13 17:00:56,448 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=false 2023-02-13 17:00:56,448 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=false 2023-02-13 17:00:56,448 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=false 2023-02-13 17:00:56,449 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=false 2023-02-13 17:00:56,449 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN 2023-02-13 17:00:56,449 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN 2023-02-13 17:00:56,449 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=false 2023-02-13 17:00:56,449 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=false 2023-02-13 17:00:56,598 INFO [jenkins-hbase4:40255] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 17:00:56,602 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:56,602 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:56,602 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307656602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307656602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307656602"}]},"ts":"1676307656602"} 2023-02-13 17:00:56,602 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:56,602 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:56,602 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:56,602 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307656602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307656602"}]},"ts":"1676307656602"} 2023-02-13 17:00:56,602 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307656602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307656602"}]},"ts":"1676307656602"} 2023-02-13 17:00:56,602 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307656602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307656602"}]},"ts":"1676307656602"} 2023-02-13 17:00:56,602 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307656602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307656602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307656602"}]},"ts":"1676307656602"} 2023-02-13 17:00:56,603 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=208, state=RUNNABLE; OpenRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:56,604 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=204, state=RUNNABLE; OpenRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:56,604 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=207, state=RUNNABLE; OpenRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:56,605 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=205, state=RUNNABLE; OpenRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:56,605 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=203, state=RUNNABLE; OpenRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:56,606 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:56,606 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656606"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307656606"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307656606"}]},"ts":"1676307656606"} 2023-02-13 17:00:56,607 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=206, state=RUNNABLE; OpenRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 17:00:56,756 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:56,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b1c0a10356cfd3754fa8161677e0dd55, NAME => 'testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 17:00:56,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:56,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:56,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d204882d59ec244668fd188df03d63c2, NAME => 'testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 17:00:56,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:56,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:56,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:56,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:56,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:56,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:56,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 14fbb78d336eb28a243b7c7310fa32d2, NAME => 'testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 17:00:56,758 INFO [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:56,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:56,758 INFO [StoreOpener-d204882d59ec244668fd188df03d63c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:56,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:56,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:56,759 INFO [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:56,759 DEBUG [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf 2023-02-13 17:00:56,759 DEBUG [StoreOpener-d204882d59ec244668fd188df03d63c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf 2023-02-13 17:00:56,759 DEBUG [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf 2023-02-13 17:00:56,759 DEBUG [StoreOpener-d204882d59ec244668fd188df03d63c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf 2023-02-13 17:00:56,760 INFO [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b1c0a10356cfd3754fa8161677e0dd55 columnFamilyName cf 2023-02-13 17:00:56,760 INFO [StoreOpener-d204882d59ec244668fd188df03d63c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d204882d59ec244668fd188df03d63c2 columnFamilyName cf 2023-02-13 17:00:56,760 DEBUG [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf 2023-02-13 17:00:56,760 DEBUG [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf 2023-02-13 17:00:56,761 INFO [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 14fbb78d336eb28a243b7c7310fa32d2 columnFamilyName cf 2023-02-13 17:00:56,761 INFO [StoreOpener-d204882d59ec244668fd188df03d63c2-1] regionserver.HStore(310): Store=d204882d59ec244668fd188df03d63c2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:56,761 INFO [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] regionserver.HStore(310): Store=14fbb78d336eb28a243b7c7310fa32d2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:56,761 INFO [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] regionserver.HStore(310): Store=b1c0a10356cfd3754fa8161677e0dd55/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:56,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:56,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:56,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:56,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:56,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:56,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:56,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:56,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:56,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:56,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:56,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:56,767 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d204882d59ec244668fd188df03d63c2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69005534, jitterRate=0.028262585401535034}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:56,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:56,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d204882d59ec244668fd188df03d63c2: Running coprocessor pre-open hook at 1676307656757Writing region info on filesystem at 1676307656757Initializing all the Stores at 1676307656758 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307656758Cleaning up temporary data from old regions at 1676307656763 (+5 ms)Cleaning up detritus from prior splits at 1676307656763Running coprocessor post-open hooks at 1676307656767 (+4 ms)Region opened successfully at 1676307656767 2023-02-13 17:00:56,768 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b1c0a10356cfd3754fa8161677e0dd55; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73771642, jitterRate=0.0992831289768219}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:56,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b1c0a10356cfd3754fa8161677e0dd55: Running coprocessor pre-open hook at 1676307656757Writing region info on filesystem at 1676307656757Initializing all the Stores at 1676307656758 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307656758Cleaning up temporary data from old regions at 1676307656763 (+5 ms)Cleaning up detritus from prior splits at 1676307656763Running coprocessor post-open hooks at 1676307656768 (+5 ms)Region opened successfully at 1676307656768 2023-02-13 17:00:56,768 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 14fbb78d336eb28a243b7c7310fa32d2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58781421, jitterRate=-0.12408857047557831}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:56,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 14fbb78d336eb28a243b7c7310fa32d2: Running coprocessor pre-open hook at 1676307656758Writing region info on filesystem at 1676307656758Initializing all the Stores at 1676307656759 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307656759Cleaning up temporary data from old regions at 1676307656763 (+4 ms)Cleaning up detritus from prior splits at 1676307656763Running coprocessor post-open hooks at 1676307656768 (+5 ms)Region opened successfully at 1676307656768 2023-02-13 17:00:56,768 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2., pid=210, masterSystemTime=1676307656755 2023-02-13 17:00:56,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2., pid=214, masterSystemTime=1676307656756 2023-02-13 17:00:56,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55., pid=209, masterSystemTime=1676307656754 2023-02-13 17:00:56,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:56,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:56,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:56,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f570113e41c274b8a0b2582dd682ff81, NAME => 'testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 17:00:56,770 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:56,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:56,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:56,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:56,770 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656770"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307656770"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307656770"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307656770"}]},"ts":"1676307656770"} 2023-02-13 17:00:56,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:56,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:56,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:56,771 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:56,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 14b9e8c5d6a0e94884bc0b364ef904bf, NAME => 'testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 17:00:56,771 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656771"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307656771"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307656771"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307656771"}]},"ts":"1676307656771"} 2023-02-13 17:00:56,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:56,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:56,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:56,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:56,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:56,771 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:56,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:56,771 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307656771"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307656771"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307656771"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307656771"}]},"ts":"1676307656771"} 2023-02-13 17:00:56,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8d87bc00cf39a8c35d32e1f7fb7f3db5, NAME => 'testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 17:00:56,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:56,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:56,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:56,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:56,772 INFO [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:56,773 INFO [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:56,773 INFO [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:56,774 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=204 2023-02-13 17:00:56,774 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=204, state=SUCCESS; OpenRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670 in 168 msec 2023-02-13 17:00:56,774 DEBUG [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf 2023-02-13 17:00:56,774 DEBUG [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf 2023-02-13 17:00:56,775 DEBUG [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf 2023-02-13 17:00:56,775 DEBUG [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf 2023-02-13 17:00:56,775 DEBUG [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf 2023-02-13 17:00:56,775 DEBUG [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf 2023-02-13 17:00:56,775 INFO [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f570113e41c274b8a0b2582dd682ff81 columnFamilyName cf 2023-02-13 17:00:56,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=206 2023-02-13 17:00:56,775 INFO [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8d87bc00cf39a8c35d32e1f7fb7f3db5 columnFamilyName cf 2023-02-13 17:00:56,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=206, state=SUCCESS; OpenRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747 in 165 msec 2023-02-13 17:00:56,775 INFO [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 14b9e8c5d6a0e94884bc0b364ef904bf columnFamilyName cf 2023-02-13 17:00:56,776 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=208 2023-02-13 17:00:56,776 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN in 329 msec 2023-02-13 17:00:56,776 INFO [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] regionserver.HStore(310): Store=f570113e41c274b8a0b2582dd682ff81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:56,776 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=208, state=SUCCESS; OpenRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716 in 170 msec 2023-02-13 17:00:56,776 INFO [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] regionserver.HStore(310): Store=8d87bc00cf39a8c35d32e1f7fb7f3db5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:56,776 INFO [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] regionserver.HStore(310): Store=14b9e8c5d6a0e94884bc0b364ef904bf/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:56,777 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN in 330 msec 2023-02-13 17:00:56,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:56,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:56,777 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN in 331 msec 2023-02-13 17:00:56,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:56,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:56,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:56,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:56,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:56,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:56,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:56,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:56,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:56,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:00:56,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f570113e41c274b8a0b2582dd682ff81; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72567553, jitterRate=0.08134080469608307}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:56,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f570113e41c274b8a0b2582dd682ff81: Running coprocessor pre-open hook at 1676307656771Writing region info on filesystem at 1676307656771Initializing all the Stores at 1676307656771Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307656771Cleaning up temporary data from old regions at 1676307656778 (+7 ms)Cleaning up detritus from prior splits at 1676307656778Running coprocessor post-open hooks at 1676307656782 (+4 ms)Region opened successfully at 1676307656782 2023-02-13 17:00:56,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 14b9e8c5d6a0e94884bc0b364ef904bf; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69892130, jitterRate=0.04147389531135559}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:56,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8d87bc00cf39a8c35d32e1f7fb7f3db5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64364480, jitterRate=-0.040894508361816406}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:56,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 14b9e8c5d6a0e94884bc0b364ef904bf: Running coprocessor pre-open hook at 1676307656771Writing region info on filesystem at 1676307656771Initializing all the Stores at 1676307656772 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307656772Cleaning up temporary data from old regions at 1676307656778 (+6 ms)Cleaning up detritus from prior splits at 1676307656779 (+1 ms)Running coprocessor post-open hooks at 1676307656782 (+3 ms)Region opened successfully at 1676307656782 2023-02-13 17:00:56,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8d87bc00cf39a8c35d32e1f7fb7f3db5: Running coprocessor pre-open hook at 1676307656772Writing region info on filesystem at 1676307656772Initializing all the Stores at 1676307656773 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307656773Cleaning up temporary data from old regions at 1676307656778 (+5 ms)Cleaning up detritus from prior splits at 1676307656778Running coprocessor post-open hooks at 1676307656782 (+4 ms)Region opened successfully at 1676307656782 2023-02-13 17:00:56,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81., pid=213, masterSystemTime=1676307656755 2023-02-13 17:00:56,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf., pid=211, masterSystemTime=1676307656756 2023-02-13 17:00:56,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5., pid=212, masterSystemTime=1676307656754 2023-02-13 17:00:56,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:56,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:56,784 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:56,784 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307656784"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307656784"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307656784"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307656784"}]},"ts":"1676307656784"} 2023-02-13 17:00:56,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:56,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:56,785 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:56,785 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656785"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307656785"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307656785"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307656785"}]},"ts":"1676307656785"} 2023-02-13 17:00:56,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:56,785 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:56,785 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:56,785 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307656785"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307656785"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307656785"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307656785"}]},"ts":"1676307656785"} 2023-02-13 17:00:56,789 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=203 2023-02-13 17:00:56,789 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=203, state=SUCCESS; OpenRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670 in 180 msec 2023-02-13 17:00:56,790 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=207 2023-02-13 17:00:56,790 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=207, state=SUCCESS; OpenRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747 in 182 msec 2023-02-13 17:00:56,790 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=205 2023-02-13 17:00:56,790 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN in 344 msec 2023-02-13 17:00:56,790 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=205, state=SUCCESS; OpenRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716 in 181 msec 2023-02-13 17:00:56,791 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN in 345 msec 2023-02-13 17:00:56,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=205, resume processing ppid=202 2023-02-13 17:00:56,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN in 345 msec 2023-02-13 17:00:56,792 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676307655966 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:00:56,792 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307656792"}]},"ts":"1676307656792"} 2023-02-13 17:00:56,793 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=ENABLED in hbase:meta 2023-02-13 17:00:56,799 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676307655966 execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:00:56,801 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676307655966 in 830 msec 2023-02-13 17:00:57,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 17:00:57,476 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676307655966, procId: 202 completed 2023-02-13 17:00:57,478 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676307655966 2023-02-13 17:00:57,478 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:57,481 INFO [Listener at localhost/40167] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:00:57,494 INFO [Listener at localhost/40167] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676307655966 2023-02-13 17:00:57,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676307655966 2023-02-13 17:00:57,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:00:57,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 17:00:57,499 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307657499"}]},"ts":"1676307657499"} 2023-02-13 17:00:57,500 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=DISABLING in hbase:meta 2023-02-13 17:00:57,502 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676307655966 to state=DISABLING 2023-02-13 17:00:57,502 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, UNASSIGN}] 2023-02-13 17:00:57,504 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, UNASSIGN 2023-02-13 17:00:57,504 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, UNASSIGN 2023-02-13 17:00:57,504 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, UNASSIGN 2023-02-13 17:00:57,504 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, UNASSIGN 2023-02-13 17:00:57,504 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, UNASSIGN 2023-02-13 17:00:57,505 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:57,505 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:57,505 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:57,505 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307657505"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307657505"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307657505"}]},"ts":"1676307657505"} 2023-02-13 17:00:57,505 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:57,505 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:57,505 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307657505"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307657505"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307657505"}]},"ts":"1676307657505"} 2023-02-13 17:00:57,506 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307657505"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307657505"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307657505"}]},"ts":"1676307657505"} 2023-02-13 17:00:57,505 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307657505"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307657505"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307657505"}]},"ts":"1676307657505"} 2023-02-13 17:00:57,505 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307657505"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307657505"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307657505"}]},"ts":"1676307657505"} 2023-02-13 17:00:57,507 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=221, state=RUNNABLE; CloseRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:57,507 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=218, state=RUNNABLE; CloseRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:57,508 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=216, state=RUNNABLE; CloseRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:57,509 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=217, state=RUNNABLE; CloseRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:57,510 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=220, state=RUNNABLE; CloseRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:57,510 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, UNASSIGN 2023-02-13 17:00:57,511 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:57,511 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307657511"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307657511"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307657511"}]},"ts":"1676307657511"} 2023-02-13 17:00:57,513 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=219, state=RUNNABLE; CloseRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:57,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:57,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b1c0a10356cfd3754fa8161677e0dd55, disabling compactions & flushes 2023-02-13 17:00:57,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:57,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:57,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. after waiting 0 ms 2023-02-13 17:00:57,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:57,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:57,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f570113e41c274b8a0b2582dd682ff81, disabling compactions & flushes 2023-02-13 17:00:57,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:57,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:57,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. after waiting 0 ms 2023-02-13 17:00:57,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:57,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:57,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:57,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 14b9e8c5d6a0e94884bc0b364ef904bf, disabling compactions & flushes 2023-02-13 17:00:57,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:57,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:57,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. after waiting 0 ms 2023-02-13 17:00:57,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:57,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:57,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b1c0a10356cfd3754fa8161677e0dd55: Waiting for close lock at 1676307657659Running coprocessor pre-close hooks at 1676307657659Disabling compacts and flushes for region at 1676307657659Disabling writes for close at 1676307657659Writing region close event to WAL at 1676307657659Running coprocessor post-close hooks at 1676307657663 (+4 ms)Closed at 1676307657663 2023-02-13 17:00:57,664 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:57,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:57,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f570113e41c274b8a0b2582dd682ff81: Waiting for close lock at 1676307657661Running coprocessor pre-close hooks at 1676307657661Disabling compacts and flushes for region at 1676307657661Disabling writes for close at 1676307657661Writing region close event to WAL at 1676307657661Running coprocessor post-close hooks at 1676307657664 (+3 ms)Closed at 1676307657664 2023-02-13 17:00:57,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:57,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:57,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 14b9e8c5d6a0e94884bc0b364ef904bf: Waiting for close lock at 1676307657663Running coprocessor pre-close hooks at 1676307657663Disabling compacts and flushes for region at 1676307657663Disabling writes for close at 1676307657663Writing region close event to WAL at 1676307657664 (+1 ms)Running coprocessor post-close hooks at 1676307657666 (+2 ms)Closed at 1676307657666 2023-02-13 17:00:57,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:57,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:57,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8d87bc00cf39a8c35d32e1f7fb7f3db5, disabling compactions & flushes 2023-02-13 17:00:57,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:57,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:57,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. after waiting 0 ms 2023-02-13 17:00:57,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:57,668 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=CLOSED 2023-02-13 17:00:57,668 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307657668"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307657668"}]},"ts":"1676307657668"} 2023-02-13 17:00:57,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:57,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:57,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d204882d59ec244668fd188df03d63c2, disabling compactions & flushes 2023-02-13 17:00:57,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:57,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:57,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. after waiting 0 ms 2023-02-13 17:00:57,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:57,669 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=CLOSED 2023-02-13 17:00:57,669 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307657669"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307657669"}]},"ts":"1676307657669"} 2023-02-13 17:00:57,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:57,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:57,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 14fbb78d336eb28a243b7c7310fa32d2, disabling compactions & flushes 2023-02-13 17:00:57,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:57,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:57,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. after waiting 0 ms 2023-02-13 17:00:57,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:57,672 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=CLOSED 2023-02-13 17:00:57,673 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307657672"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307657672"}]},"ts":"1676307657672"} 2023-02-13 17:00:57,674 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=221 2023-02-13 17:00:57,674 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=216 2023-02-13 17:00:57,674 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=221, state=SUCCESS; CloseRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716 in 163 msec 2023-02-13 17:00:57,674 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=216, state=SUCCESS; CloseRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670 in 162 msec 2023-02-13 17:00:57,675 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, UNASSIGN in 172 msec 2023-02-13 17:00:57,675 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, UNASSIGN in 172 msec 2023-02-13 17:00:57,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:57,677 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=220 2023-02-13 17:00:57,677 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=220, state=SUCCESS; CloseRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747 in 164 msec 2023-02-13 17:00:57,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:57,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8d87bc00cf39a8c35d32e1f7fb7f3db5: Waiting for close lock at 1676307657668Running coprocessor pre-close hooks at 1676307657668Disabling compacts and flushes for region at 1676307657668Disabling writes for close at 1676307657668Writing region close event to WAL at 1676307657668Running coprocessor post-close hooks at 1676307657677 (+9 ms)Closed at 1676307657677 2023-02-13 17:00:57,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:57,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:57,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d204882d59ec244668fd188df03d63c2: Waiting for close lock at 1676307657668Running coprocessor pre-close hooks at 1676307657668Disabling compacts and flushes for region at 1676307657668Disabling writes for close at 1676307657668Writing region close event to WAL at 1676307657669 (+1 ms)Running coprocessor post-close hooks at 1676307657678 (+9 ms)Closed at 1676307657678 2023-02-13 17:00:57,678 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, UNASSIGN in 175 msec 2023-02-13 17:00:57,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:57,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:00:57,679 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=CLOSED 2023-02-13 17:00:57,679 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307657679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307657679"}]},"ts":"1676307657679"} 2023-02-13 17:00:57,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:57,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 14fbb78d336eb28a243b7c7310fa32d2: Waiting for close lock at 1676307657672Running coprocessor pre-close hooks at 1676307657672Disabling compacts and flushes for region at 1676307657672Disabling writes for close at 1676307657672Writing region close event to WAL at 1676307657673 (+1 ms)Running coprocessor post-close hooks at 1676307657679 (+6 ms)Closed at 1676307657679 2023-02-13 17:00:57,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:57,681 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=CLOSED 2023-02-13 17:00:57,681 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307657681"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307657681"}]},"ts":"1676307657681"} 2023-02-13 17:00:57,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:57,682 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=CLOSED 2023-02-13 17:00:57,682 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307657682"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307657682"}]},"ts":"1676307657682"} 2023-02-13 17:00:57,683 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=218 2023-02-13 17:00:57,683 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=218, state=SUCCESS; CloseRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716 in 173 msec 2023-02-13 17:00:57,684 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=217 2023-02-13 17:00:57,684 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=217, state=SUCCESS; CloseRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670 in 173 msec 2023-02-13 17:00:57,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, UNASSIGN in 181 msec 2023-02-13 17:00:57,685 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=219 2023-02-13 17:00:57,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, UNASSIGN in 182 msec 2023-02-13 17:00:57,685 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=219, state=SUCCESS; CloseRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747 in 170 msec 2023-02-13 17:00:57,687 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=219, resume processing ppid=215 2023-02-13 17:00:57,687 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, UNASSIGN in 183 msec 2023-02-13 17:00:57,687 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307657687"}]},"ts":"1676307657687"} 2023-02-13 17:00:57,689 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=DISABLED in hbase:meta 2023-02-13 17:00:57,690 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676307655966 to state=DISABLED 2023-02-13 17:00:57,692 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676307655966 in 197 msec 2023-02-13 17:00:57,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 17:00:57,750 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676307655966, procId: 215 completed 2023-02-13 17:00:57,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 } 2023-02-13 17:00:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676307657751 (current time:1676307657751). 2023-02-13 17:00:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 17:00:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(139): Connect 0x6568d269 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:57,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@df420f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:57,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:57,760 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53710, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:57,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6568d269 to 127.0.0.1:54083 2023-02-13 17:00:57,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:57,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 17:00:57,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 } 2023-02-13 17:00:57,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676307655966, type=EXCLUSIVE 2023-02-13 17:00:57,764 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676307655966, type=EXCLUSIVE 2023-02-13 17:00:57,765 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676307655966, type=EXCLUSIVE; timeout=600000, timestamp=1676308257765 2023-02-13 17:00:57,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 } 2023-02-13 17:00:57,765 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676307655966 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676307655966 2023-02-13 17:00:57,766 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 17:00:57,766 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 17:00:57,780 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676307655966 table=testCorruptedSnapshot-1676307655966 type=DISABLED ttl=0 } 2023-02-13 17:00:57,785 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:57,785 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:57,785 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:57,785 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:57,788 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:57,788 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:57,788 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:57,788 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:57,788 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:57,788 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:57,789 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:57,789 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:57,789 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:57,790 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:57,790 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:57,790 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:57,791 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:57,792 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:57,792 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:57,792 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:57,793 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 17:00:57,855 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676307655966 table=testCorruptedSnapshot-1676307655966 type=DISABLED ttl=0 } as finished. 2023-02-13 17:00:57,855 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676307655966 2023-02-13 17:00:57,855 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/emptySnaptb-1676307655966 2023-02-13 17:00:57,878 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/emptySnaptb-1676307655966 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/emptySnaptb-1676307655966 2023-02-13 17:00:57,894 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676307655966 of table testCorruptedSnapshot-1676307655966 completed 2023-02-13 17:00:57,894 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/emptySnaptb-1676307655966 2023-02-13 17:00:57,894 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/emptySnaptb-1676307655966 2023-02-13 17:00:57,894 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676307655966 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676307655966 at 1676307657765Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676307655966 table=testCorruptedSnapshot-1676307655966 type=DISABLED ttl=0 } at 1676307657780 (+15 ms)Consolidate snapshot: emptySnaptb-1676307655966 at 1676307657855 (+75 ms)Loading Region manifests for emptySnaptb-1676307655966 at 1676307657855Writing data manifest for emptySnaptb-1676307655966 at 1676307657859 (+4 ms)Verifying snapshot: emptySnaptb-1676307655966 at 1676307657873 (+14 ms)Snapshot emptySnaptb-1676307655966 of table testCorruptedSnapshot-1676307655966 completed at 1676307657894 (+21 ms) 2023-02-13 17:00:57,896 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676307655966, type=EXCLUSIVE 2023-02-13 17:00:57,897 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676307655966, type=EXCLUSIVE in 133 msec 2023-02-13 17:00:58,016 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 17:00:58,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 } is done 2023-02-13 17:00:58,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 17:00:58,018 INFO [Listener at localhost/40167] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676307655966 2023-02-13 17:00:58,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676307655966 2023-02-13 17:00:58,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:00:58,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 17:00:58,022 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676307655966 2023-02-13 17:00:58,022 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307658022"}]},"ts":"1676307658022"} 2023-02-13 17:00:58,023 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=ENABLING in hbase:meta 2023-02-13 17:00:58,025 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 17:00:58,026 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN}] 2023-02-13 17:00:58,029 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN 2023-02-13 17:00:58,029 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN 2023-02-13 17:00:58,029 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN 2023-02-13 17:00:58,029 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN 2023-02-13 17:00:58,030 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:58,030 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:58,030 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=true 2023-02-13 17:00:58,030 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=true 2023-02-13 17:00:58,030 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:58,030 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=true 2023-02-13 17:00:58,030 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN 2023-02-13 17:00:58,030 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:58,030 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN 2023-02-13 17:00:58,030 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=true 2023-02-13 17:00:58,031 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:58,031 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=true 2023-02-13 17:00:58,031 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:58,031 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=true 2023-02-13 17:00:58,180 INFO [jenkins-hbase4:40255] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 17:00:58,180 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:58,180 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:58,181 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307658180"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658180"}]},"ts":"1676307658180"} 2023-02-13 17:00:58,180 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:58,180 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:58,181 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658180"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658180"}]},"ts":"1676307658180"} 2023-02-13 17:00:58,181 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307658180"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658180"}]},"ts":"1676307658180"} 2023-02-13 17:00:58,181 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658180"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658180"}]},"ts":"1676307658180"} 2023-02-13 17:00:58,180 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:58,181 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658180"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658180"}]},"ts":"1676307658180"} 2023-02-13 17:00:58,182 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=230, state=RUNNABLE; OpenRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:58,183 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=231, state=RUNNABLE; OpenRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:58,184 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=235, state=RUNNABLE; OpenRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:58,184 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=234, state=RUNNABLE; OpenRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:58,185 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=232, state=RUNNABLE; OpenRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:58,185 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:58,185 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658185"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658185"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658185"}]},"ts":"1676307658185"} 2023-02-13 17:00:58,186 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=233, state=RUNNABLE; OpenRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:58,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 17:00:58,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:58,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f570113e41c274b8a0b2582dd682ff81, NAME => 'testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 17:00:58,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:58,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:58,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:58,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:58,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:58,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:58,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b1c0a10356cfd3754fa8161677e0dd55, NAME => 'testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 17:00:58,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 14b9e8c5d6a0e94884bc0b364ef904bf, NAME => 'testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 17:00:58,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:58,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:58,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:58,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:58,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:58,338 INFO [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:58,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:58,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:58,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:58,339 INFO [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:58,340 DEBUG [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf 2023-02-13 17:00:58,340 DEBUG [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf 2023-02-13 17:00:58,340 INFO [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:58,340 INFO [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f570113e41c274b8a0b2582dd682ff81 columnFamilyName cf 2023-02-13 17:00:58,340 DEBUG [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf 2023-02-13 17:00:58,340 DEBUG [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf 2023-02-13 17:00:58,341 INFO [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b1c0a10356cfd3754fa8161677e0dd55 columnFamilyName cf 2023-02-13 17:00:58,341 DEBUG [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf 2023-02-13 17:00:58,341 DEBUG [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf 2023-02-13 17:00:58,341 INFO [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] regionserver.HStore(310): Store=f570113e41c274b8a0b2582dd682ff81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:58,341 INFO [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 14b9e8c5d6a0e94884bc0b364ef904bf columnFamilyName cf 2023-02-13 17:00:58,341 INFO [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] regionserver.HStore(310): Store=b1c0a10356cfd3754fa8161677e0dd55/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:58,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:58,342 INFO [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] regionserver.HStore(310): Store=14b9e8c5d6a0e94884bc0b364ef904bf/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:58,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:58,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:58,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:58,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:58,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:58,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:58,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:58,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:58,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b1c0a10356cfd3754fa8161677e0dd55; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64120075, jitterRate=-0.04453642666339874}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:58,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f570113e41c274b8a0b2582dd682ff81; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66145060, jitterRate=-0.014361798763275146}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:58,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b1c0a10356cfd3754fa8161677e0dd55: Running coprocessor pre-open hook at 1676307658338Writing region info on filesystem at 1676307658338Initializing all the Stores at 1676307658339 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307658339Cleaning up temporary data from old regions at 1676307658344 (+5 ms)Cleaning up detritus from prior splits at 1676307658344Running coprocessor post-open hooks at 1676307658346 (+2 ms)Region opened successfully at 1676307658346 2023-02-13 17:00:58,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f570113e41c274b8a0b2582dd682ff81: Running coprocessor pre-open hook at 1676307658337Writing region info on filesystem at 1676307658337Initializing all the Stores at 1676307658337Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307658337Cleaning up temporary data from old regions at 1676307658343 (+6 ms)Cleaning up detritus from prior splits at 1676307658343Running coprocessor post-open hooks at 1676307658346 (+3 ms)Region opened successfully at 1676307658346 2023-02-13 17:00:58,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 14b9e8c5d6a0e94884bc0b364ef904bf; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70973317, jitterRate=0.05758483707904816}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:58,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 14b9e8c5d6a0e94884bc0b364ef904bf: Running coprocessor pre-open hook at 1676307658338Writing region info on filesystem at 1676307658338Initializing all the Stores at 1676307658339 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307658340 (+1 ms)Cleaning up temporary data from old regions at 1676307658344 (+4 ms)Cleaning up detritus from prior splits at 1676307658344Running coprocessor post-open hooks at 1676307658347 (+3 ms)Region opened successfully at 1676307658347 2023-02-13 17:00:58,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55., pid=238, masterSystemTime=1676307658335 2023-02-13 17:00:58,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81., pid=236, masterSystemTime=1676307658334 2023-02-13 17:00:58,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf., pid=239, masterSystemTime=1676307658335 2023-02-13 17:00:58,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:58,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:58,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:58,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8d87bc00cf39a8c35d32e1f7fb7f3db5, NAME => 'testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 17:00:58,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:58,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:58,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:58,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:58,349 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:58,349 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307658349"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307658349"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307658349"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307658349"}]},"ts":"1676307658349"} 2023-02-13 17:00:58,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:58,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:58,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:58,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d204882d59ec244668fd188df03d63c2, NAME => 'testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 17:00:58,350 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:58,350 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307658350"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307658350"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307658350"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307658350"}]},"ts":"1676307658350"} 2023-02-13 17:00:58,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:58,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:58,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:58,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:58,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:58,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:58,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 14fbb78d336eb28a243b7c7310fa32d2, NAME => 'testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 17:00:58,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:58,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:58,351 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:58,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:00:58,351 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658351"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307658351"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307658351"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307658351"}]},"ts":"1676307658351"} 2023-02-13 17:00:58,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:58,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:58,351 INFO [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:58,352 INFO [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:58,353 DEBUG [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf 2023-02-13 17:00:58,353 DEBUG [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf 2023-02-13 17:00:58,353 INFO [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8d87bc00cf39a8c35d32e1f7fb7f3db5 columnFamilyName cf 2023-02-13 17:00:58,354 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=235 2023-02-13 17:00:58,354 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=235, state=SUCCESS; OpenRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716 in 168 msec 2023-02-13 17:00:58,354 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=230 2023-02-13 17:00:58,354 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=230, state=SUCCESS; OpenRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670 in 170 msec 2023-02-13 17:00:58,354 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=234 2023-02-13 17:00:58,355 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=234, state=SUCCESS; OpenRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747 in 168 msec 2023-02-13 17:00:58,355 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN in 328 msec 2023-02-13 17:00:58,356 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN in 328 msec 2023-02-13 17:00:58,356 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN in 328 msec 2023-02-13 17:00:58,359 DEBUG [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf 2023-02-13 17:00:58,359 DEBUG [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf 2023-02-13 17:00:58,359 INFO [StoreOpener-d204882d59ec244668fd188df03d63c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:58,360 INFO [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 14fbb78d336eb28a243b7c7310fa32d2 columnFamilyName cf 2023-02-13 17:00:58,361 INFO [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] regionserver.HStore(310): Store=8d87bc00cf39a8c35d32e1f7fb7f3db5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:58,361 INFO [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] regionserver.HStore(310): Store=14fbb78d336eb28a243b7c7310fa32d2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:58,361 DEBUG [StoreOpener-d204882d59ec244668fd188df03d63c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf 2023-02-13 17:00:58,362 DEBUG [StoreOpener-d204882d59ec244668fd188df03d63c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf 2023-02-13 17:00:58,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:58,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:58,362 INFO [StoreOpener-d204882d59ec244668fd188df03d63c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d204882d59ec244668fd188df03d63c2 columnFamilyName cf 2023-02-13 17:00:58,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:58,363 INFO [StoreOpener-d204882d59ec244668fd188df03d63c2-1] regionserver.HStore(310): Store=d204882d59ec244668fd188df03d63c2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:00:58,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:58,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:58,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:58,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:58,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:58,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 14fbb78d336eb28a243b7c7310fa32d2; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71497881, jitterRate=0.06540144979953766}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:58,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 14fbb78d336eb28a243b7c7310fa32d2: Running coprocessor pre-open hook at 1676307658351Writing region info on filesystem at 1676307658351Initializing all the Stores at 1676307658352 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307658352Cleaning up temporary data from old regions at 1676307658363 (+11 ms)Cleaning up detritus from prior splits at 1676307658364 (+1 ms)Running coprocessor post-open hooks at 1676307658366 (+2 ms)Region opened successfully at 1676307658366 2023-02-13 17:00:58,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8d87bc00cf39a8c35d32e1f7fb7f3db5; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74729549, jitterRate=0.11355705559253693}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:58,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8d87bc00cf39a8c35d32e1f7fb7f3db5: Running coprocessor pre-open hook at 1676307658349Writing region info on filesystem at 1676307658349Initializing all the Stores at 1676307658350 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307658350Cleaning up temporary data from old regions at 1676307658363 (+13 ms)Cleaning up detritus from prior splits at 1676307658364 (+1 ms)Running coprocessor post-open hooks at 1676307658366 (+2 ms)Region opened successfully at 1676307658366 2023-02-13 17:00:58,367 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2., pid=241, masterSystemTime=1676307658335 2023-02-13 17:00:58,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:58,367 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5., pid=240, masterSystemTime=1676307658335 2023-02-13 17:00:58,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d204882d59ec244668fd188df03d63c2; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63971285, jitterRate=-0.046753570437431335}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:00:58,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d204882d59ec244668fd188df03d63c2: Running coprocessor pre-open hook at 1676307658351Writing region info on filesystem at 1676307658351Initializing all the Stores at 1676307658352 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307658352Cleaning up temporary data from old regions at 1676307658365 (+13 ms)Cleaning up detritus from prior splits at 1676307658365Running coprocessor post-open hooks at 1676307658368 (+3 ms)Region opened successfully at 1676307658368 2023-02-13 17:00:58,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:58,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:58,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2., pid=237, masterSystemTime=1676307658334 2023-02-13 17:00:58,369 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:58,369 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658368"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307658368"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307658368"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307658368"}]},"ts":"1676307658368"} 2023-02-13 17:00:58,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:58,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:58,369 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:58,370 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658369"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307658369"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307658369"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307658369"}]},"ts":"1676307658369"} 2023-02-13 17:00:58,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:58,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:58,370 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:58,371 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658370"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307658370"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307658370"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307658370"}]},"ts":"1676307658370"} 2023-02-13 17:00:58,372 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=233 2023-02-13 17:00:58,372 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=233, state=SUCCESS; OpenRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747 in 184 msec 2023-02-13 17:00:58,373 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=232 2023-02-13 17:00:58,373 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=232, state=SUCCESS; OpenRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716 in 186 msec 2023-02-13 17:00:58,373 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN in 346 msec 2023-02-13 17:00:58,374 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=231 2023-02-13 17:00:58,374 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=231, state=SUCCESS; OpenRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670 in 189 msec 2023-02-13 17:00:58,374 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN in 347 msec 2023-02-13 17:00:58,375 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=231, resume processing ppid=229 2023-02-13 17:00:58,375 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN in 348 msec 2023-02-13 17:00:58,376 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307658376"}]},"ts":"1676307658376"} 2023-02-13 17:00:58,377 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=ENABLED in hbase:meta 2023-02-13 17:00:58,378 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676307655966' was successfully enabled. 2023-02-13 17:00:58,379 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676307655966 in 361 msec 2023-02-13 17:00:58,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 17:00:58,774 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676307655966, procId: 229 completed 2023-02-13 17:00:58,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:58,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:58,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:58,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:58,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:58,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:00:58,789 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676307655966 2023-02-13 17:00:58,789 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:58,790 INFO [Listener at localhost/40167] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:00:58,801 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 17:00:58,802 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 17:00:58,803 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 17:00:58,804 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 17:00:58,805 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 17:00:58,808 INFO [Listener at localhost/40167] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676307655966 2023-02-13 17:00:58,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676307655966 2023-02-13 17:00:58,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:00:58,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 17:00:58,812 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307658812"}]},"ts":"1676307658812"} 2023-02-13 17:00:58,813 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=DISABLING in hbase:meta 2023-02-13 17:00:58,815 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676307655966 to state=DISABLING 2023-02-13 17:00:58,816 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, UNASSIGN}] 2023-02-13 17:00:58,817 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, UNASSIGN 2023-02-13 17:00:58,817 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, UNASSIGN 2023-02-13 17:00:58,817 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, UNASSIGN 2023-02-13 17:00:58,817 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, UNASSIGN 2023-02-13 17:00:58,817 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, UNASSIGN 2023-02-13 17:00:58,818 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:58,818 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:58,818 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658817"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658817"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658817"}]},"ts":"1676307658817"} 2023-02-13 17:00:58,818 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658818"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658818"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658818"}]},"ts":"1676307658818"} 2023-02-13 17:00:58,818 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:58,818 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307658818"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658818"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658818"}]},"ts":"1676307658818"} 2023-02-13 17:00:58,818 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:58,818 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:58,818 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658818"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658818"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658818"}]},"ts":"1676307658818"} 2023-02-13 17:00:58,819 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307658818"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658818"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658818"}]},"ts":"1676307658818"} 2023-02-13 17:00:58,820 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=247, state=RUNNABLE; CloseRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:58,821 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=245, state=RUNNABLE; CloseRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:58,821 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=248, state=RUNNABLE; CloseRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:58,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=246, state=RUNNABLE; CloseRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:58,823 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=244, state=RUNNABLE; CloseRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:58,824 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, UNASSIGN 2023-02-13 17:00:58,825 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:58,825 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307658825"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307658825"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307658825"}]},"ts":"1676307658825"} 2023-02-13 17:00:58,826 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=243, state=RUNNABLE; CloseRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:58,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:58,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 14fbb78d336eb28a243b7c7310fa32d2, disabling compactions & flushes 2023-02-13 17:00:58,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:58,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:58,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. after waiting 0 ms 2023-02-13 17:00:58,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:58,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 14fbb78d336eb28a243b7c7310fa32d2 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-13 17:00:58,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:58,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b1c0a10356cfd3754fa8161677e0dd55, disabling compactions & flushes 2023-02-13 17:00:58,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:58,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:58,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. after waiting 0 ms 2023-02-13 17:00:58,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:58,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing b1c0a10356cfd3754fa8161677e0dd55 1/1 column families, dataSize=22.70 KB heapSize=49.05 KB 2023-02-13 17:00:58,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:58,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d204882d59ec244668fd188df03d63c2, disabling compactions & flushes 2023-02-13 17:00:58,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:58,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:58,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. after waiting 0 ms 2023-02-13 17:00:58,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:58,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d204882d59ec244668fd188df03d63c2 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-13 17:00:59,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132684611ba22b4619bbcce782b82ecb0a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132684611ba22b4619bbcce782b82ecb0a 2023-02-13 17:00:59,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302136ec2d881e3254ea59a655def7005a638 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/e4da3b7fbbce2345d7772b0674a318d5202302136ec2d881e3254ea59a655def7005a638 2023-02-13 17:00:59,014 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213f0259ef983bd4d8db5e54997e978832c to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c4ca4238a0b923820dcc509a6f75849b20230213f0259ef983bd4d8db5e54997e978832c 2023-02-13 17:00:59,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/.tmp/cf/a6e630d19d804cff8dfab678dc0b4a69 2023-02-13 17:00:59,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/.tmp/cf/a6e630d19d804cff8dfab678dc0b4a69 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/a6e630d19d804cff8dfab678dc0b4a69 2023-02-13 17:00:59,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/.tmp/cf/6b5e35109ff44629bc962aa39bd36925 2023-02-13 17:00:59,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/a6e630d19d804cff8dfab678dc0b4a69, entries=347, sequenceid=8, filesize=64.7 K 2023-02-13 17:00:59,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.70 KB/23245, heapSize ~49.03 KB/50208, currentSize=0 B/0 for b1c0a10356cfd3754fa8161677e0dd55 in 64ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:59,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676307655966' 2023-02-13 17:00:59,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/.tmp/cf/6b5e35109ff44629bc962aa39bd36925 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/6b5e35109ff44629bc962aa39bd36925 2023-02-13 17:00:59,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/.tmp/cf/93866f2c0f744934afb3530f0a45adc6 2023-02-13 17:00:59,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/6b5e35109ff44629bc962aa39bd36925, entries=28, sequenceid=8, filesize=9.7 K 2023-02-13 17:00:59,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:59,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/.tmp/cf/93866f2c0f744934afb3530f0a45adc6 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/93866f2c0f744934afb3530f0a45adc6 2023-02-13 17:00:59,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 14fbb78d336eb28a243b7c7310fa32d2 in 71ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:59,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:00:59,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b1c0a10356cfd3754fa8161677e0dd55: Waiting for close lock at 1676307658975Running coprocessor pre-close hooks at 1676307658975Disabling compacts and flushes for region at 1676307658975Disabling writes for close at 1676307658975Obtaining lock to block concurrent updates at 1676307658975Preparing flush snapshotting stores in b1c0a10356cfd3754fa8161677e0dd55 at 1676307658975Finished memstore snapshotting testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55., syncing WAL and waiting on mvcc, flushsize=dataSize=23245, getHeapSize=50208, getOffHeapSize=0, getCellsCount=347 at 1676307658975Flushing stores of testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. at 1676307658976 (+1 ms)Flushing b1c0a10356cfd3754fa8161677e0dd55/cf: creating writer at 1676307658977 (+1 ms)Flushing mob file b1c0a10356cfd3754fa8161677e0dd55/cf: appending metadata at 1676307658983 (+6 ms)Flushing mob file b1c0a10356cfd3754fa8161677e0dd55/cf: closing flushed file at 1676307658983Flushing b1c0a10356cfd3754fa8161677e0dd55/cf: appending metadata at 1676307659009 (+26 ms)Flushing b1c0a10356cfd3754fa8161677e0dd55/cf: closing flushed file at 1676307659009Flushing b1c0a10356cfd3754fa8161677e0dd55/cf: reopening flushed file at 1676307659035 (+26 ms)Finished flush of dataSize ~22.70 KB/23245, heapSize ~49.03 KB/50208, currentSize=0 B/0 for b1c0a10356cfd3754fa8161677e0dd55 in 64ms, sequenceid=8, compaction requested=false at 1676307659039 (+4 ms)Writing region close event to WAL at 1676307659040 (+1 ms)Running coprocessor post-close hooks at 1676307659045 (+5 ms)Closed at 1676307659045 2023-02-13 17:00:59,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:00:59,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:59,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8d87bc00cf39a8c35d32e1f7fb7f3db5, disabling compactions & flushes 2023-02-13 17:00:59,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:59,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:59,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. after waiting 0 ms 2023-02-13 17:00:59,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:59,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 8d87bc00cf39a8c35d32e1f7fb7f3db5 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-13 17:00:59,046 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=CLOSED 2023-02-13 17:00:59,047 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307659046"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307659046"}]},"ts":"1676307659046"} 2023-02-13 17:00:59,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:59,049 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:00:59,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 14fbb78d336eb28a243b7c7310fa32d2: Waiting for close lock at 1676307658972Running coprocessor pre-close hooks at 1676307658972Disabling compacts and flushes for region at 1676307658972Disabling writes for close at 1676307658972Obtaining lock to block concurrent updates at 1676307658972Preparing flush snapshotting stores in 14fbb78d336eb28a243b7c7310fa32d2 at 1676307658972Finished memstore snapshotting testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676307658973 (+1 ms)Flushing stores of testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. at 1676307658973Flushing 14fbb78d336eb28a243b7c7310fa32d2/cf: creating writer at 1676307658974 (+1 ms)Flushing mob file 14fbb78d336eb28a243b7c7310fa32d2/cf: appending metadata at 1676307658981 (+7 ms)Flushing mob file 14fbb78d336eb28a243b7c7310fa32d2/cf: closing flushed file at 1676307658981Flushing 14fbb78d336eb28a243b7c7310fa32d2/cf: appending metadata at 1676307659009 (+28 ms)Flushing 14fbb78d336eb28a243b7c7310fa32d2/cf: closing flushed file at 1676307659009Flushing 14fbb78d336eb28a243b7c7310fa32d2/cf: reopening flushed file at 1676307659040 (+31 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 14fbb78d336eb28a243b7c7310fa32d2 in 71ms, sequenceid=8, compaction requested=false at 1676307659043 (+3 ms)Writing region close event to WAL at 1676307659045 (+2 ms)Running coprocessor post-close hooks at 1676307659049 (+4 ms)Closed at 1676307659049 2023-02-13 17:00:59,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/93866f2c0f744934afb3530f0a45adc6, entries=26, sequenceid=8, filesize=9.4 K 2023-02-13 17:00:59,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for d204882d59ec244668fd188df03d63c2 in 73ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:59,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:00:59,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:59,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 14b9e8c5d6a0e94884bc0b364ef904bf, disabling compactions & flushes 2023-02-13 17:00:59,052 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=CLOSED 2023-02-13 17:00:59,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:59,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:59,052 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307659052"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307659052"}]},"ts":"1676307659052"} 2023-02-13 17:00:59,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. after waiting 0 ms 2023-02-13 17:00:59,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:59,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 14b9e8c5d6a0e94884bc0b364ef904bf 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-13 17:00:59,054 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=248 2023-02-13 17:00:59,054 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=248, state=SUCCESS; CloseRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716 in 228 msec 2023-02-13 17:00:59,057 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, UNASSIGN in 238 msec 2023-02-13 17:00:59,059 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=246 2023-02-13 17:00:59,059 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=246, state=SUCCESS; CloseRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747 in 236 msec 2023-02-13 17:00:59,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:59,060 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, UNASSIGN in 243 msec 2023-02-13 17:00:59,061 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:00:59,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d204882d59ec244668fd188df03d63c2: Waiting for close lock at 1676307658977Running coprocessor pre-close hooks at 1676307658977Disabling compacts and flushes for region at 1676307658977Disabling writes for close at 1676307658977Obtaining lock to block concurrent updates at 1676307658977Preparing flush snapshotting stores in d204882d59ec244668fd188df03d63c2 at 1676307658977Finished memstore snapshotting testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676307658977Flushing stores of testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. at 1676307658978 (+1 ms)Flushing d204882d59ec244668fd188df03d63c2/cf: creating writer at 1676307658979 (+1 ms)Flushing mob file d204882d59ec244668fd188df03d63c2/cf: appending metadata at 1676307658984 (+5 ms)Flushing mob file d204882d59ec244668fd188df03d63c2/cf: closing flushed file at 1676307658984Flushing d204882d59ec244668fd188df03d63c2/cf: appending metadata at 1676307659015 (+31 ms)Flushing d204882d59ec244668fd188df03d63c2/cf: closing flushed file at 1676307659015Flushing d204882d59ec244668fd188df03d63c2/cf: reopening flushed file at 1676307659045 (+30 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for d204882d59ec244668fd188df03d63c2 in 73ms, sequenceid=8, compaction requested=false at 1676307659050 (+5 ms)Writing region close event to WAL at 1676307659052 (+2 ms)Running coprocessor post-close hooks at 1676307659061 (+9 ms)Closed at 1676307659061 2023-02-13 17:00:59,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d204882d59ec244668fd188df03d63c2 2023-02-13 17:00:59,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:59,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f570113e41c274b8a0b2582dd682ff81, disabling compactions & flushes 2023-02-13 17:00:59,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:59,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:59,064 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. after waiting 0 ms 2023-02-13 17:00:59,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 17:00:59,064 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=CLOSED 2023-02-13 17:00:59,064 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307659064"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307659064"}]},"ts":"1676307659064"} 2023-02-13 17:00:59,064 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:59,064 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f570113e41c274b8a0b2582dd682ff81 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-13 17:00:59,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302135ce47f848d6f4a4198706abbf241a932 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c81e728d9d4c2f636f067f89cc14862c202302135ce47f848d6f4a4198706abbf241a932 2023-02-13 17:00:59,067 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=244 2023-02-13 17:00:59,067 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=244, state=SUCCESS; CloseRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670 in 242 msec 2023-02-13 17:00:59,068 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, UNASSIGN in 251 msec 2023-02-13 17:00:59,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/.tmp/cf/29756e1075774bad95cd552f57055e05 2023-02-13 17:00:59,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213476fe5d0115c4dadb854b4679cc317db to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/a87ff679a2f3e71d9181a67b7542122c20230213476fe5d0115c4dadb854b4679cc317db 2023-02-13 17:00:59,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302130114bb04302148cf97fc307faf95443d to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/d41d8cd98f00b204e9800998ecf8427e202302130114bb04302148cf97fc307faf95443d 2023-02-13 17:00:59,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/.tmp/cf/29756e1075774bad95cd552f57055e05 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/29756e1075774bad95cd552f57055e05 2023-02-13 17:00:59,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/29756e1075774bad95cd552f57055e05, entries=39, sequenceid=8, filesize=11.6 K 2023-02-13 17:00:59,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 8d87bc00cf39a8c35d32e1f7fb7f3db5 in 44ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:59,092 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/.tmp/cf/be910e1657a440b58fb8e57ec0573d5c 2023-02-13 17:00:59,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/.tmp/cf/eb324649c13e4af88dbc2f2d549860e4 2023-02-13 17:00:59,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/.tmp/cf/be910e1657a440b58fb8e57ec0573d5c as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c 2023-02-13 17:00:59,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:59,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:00:59,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8d87bc00cf39a8c35d32e1f7fb7f3db5: Waiting for close lock at 1676307659046Running coprocessor pre-close hooks at 1676307659046Disabling compacts and flushes for region at 1676307659046Disabling writes for close at 1676307659046Obtaining lock to block concurrent updates at 1676307659046Preparing flush snapshotting stores in 8d87bc00cf39a8c35d32e1f7fb7f3db5 at 1676307659046Finished memstore snapshotting testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676307659047 (+1 ms)Flushing stores of testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. at 1676307659047Flushing 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: creating writer at 1676307659048 (+1 ms)Flushing mob file 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: appending metadata at 1676307659052 (+4 ms)Flushing mob file 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: closing flushed file at 1676307659052Flushing 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: appending metadata at 1676307659067 (+15 ms)Flushing 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: closing flushed file at 1676307659068 (+1 ms)Flushing 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: reopening flushed file at 1676307659086 (+18 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 8d87bc00cf39a8c35d32e1f7fb7f3db5 in 44ms, sequenceid=8, compaction requested=false at 1676307659090 (+4 ms)Writing region close event to WAL at 1676307659092 (+2 ms)Running coprocessor post-close hooks at 1676307659096 (+4 ms)Closed at 1676307659096 2023-02-13 17:00:59,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:00:59,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/.tmp/cf/eb324649c13e4af88dbc2f2d549860e4 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/eb324649c13e4af88dbc2f2d549860e4 2023-02-13 17:00:59,099 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=CLOSED 2023-02-13 17:00:59,099 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307659099"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307659099"}]},"ts":"1676307659099"} 2023-02-13 17:00:59,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c, entries=27, sequenceid=8, filesize=9.6 K 2023-02-13 17:00:59,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 14b9e8c5d6a0e94884bc0b364ef904bf in 48ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:59,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/eb324649c13e4af88dbc2f2d549860e4, entries=33, sequenceid=8, filesize=10.6 K 2023-02-13 17:00:59,102 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=245 2023-02-13 17:00:59,102 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=245, state=SUCCESS; CloseRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716 in 279 msec 2023-02-13 17:00:59,103 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for f570113e41c274b8a0b2582dd682ff81 in 39ms, sequenceid=8, compaction requested=false 2023-02-13 17:00:59,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:59,103 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, UNASSIGN in 286 msec 2023-02-13 17:00:59,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:00:59,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 14b9e8c5d6a0e94884bc0b364ef904bf: Waiting for close lock at 1676307659052Running coprocessor pre-close hooks at 1676307659052Disabling compacts and flushes for region at 1676307659052Disabling writes for close at 1676307659052Obtaining lock to block concurrent updates at 1676307659053 (+1 ms)Preparing flush snapshotting stores in 14b9e8c5d6a0e94884bc0b364ef904bf at 1676307659053Finished memstore snapshotting testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676307659053Flushing stores of testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. at 1676307659054 (+1 ms)Flushing 14b9e8c5d6a0e94884bc0b364ef904bf/cf: creating writer at 1676307659056 (+2 ms)Flushing mob file 14b9e8c5d6a0e94884bc0b364ef904bf/cf: appending metadata at 1676307659061 (+5 ms)Flushing mob file 14b9e8c5d6a0e94884bc0b364ef904bf/cf: closing flushed file at 1676307659061Flushing 14b9e8c5d6a0e94884bc0b364ef904bf/cf: appending metadata at 1676307659084 (+23 ms)Flushing 14b9e8c5d6a0e94884bc0b364ef904bf/cf: closing flushed file at 1676307659084Flushing 14b9e8c5d6a0e94884bc0b364ef904bf/cf: reopening flushed file at 1676307659096 (+12 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 14b9e8c5d6a0e94884bc0b364ef904bf in 48ms, sequenceid=8, compaction requested=false at 1676307659100 (+4 ms)Writing region close event to WAL at 1676307659100Running coprocessor post-close hooks at 1676307659104 (+4 ms)Closed at 1676307659104 2023-02-13 17:00:59,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:00:59,105 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=CLOSED 2023-02-13 17:00:59,105 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307659105"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307659105"}]},"ts":"1676307659105"} 2023-02-13 17:00:59,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 17:00:59,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:00:59,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f570113e41c274b8a0b2582dd682ff81: Waiting for close lock at 1676307659063Running coprocessor pre-close hooks at 1676307659063Disabling compacts and flushes for region at 1676307659063Disabling writes for close at 1676307659064 (+1 ms)Obtaining lock to block concurrent updates at 1676307659064Preparing flush snapshotting stores in f570113e41c274b8a0b2582dd682ff81 at 1676307659064Finished memstore snapshotting testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676307659064Flushing stores of testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. at 1676307659065 (+1 ms)Flushing f570113e41c274b8a0b2582dd682ff81/cf: creating writer at 1676307659065Flushing mob file f570113e41c274b8a0b2582dd682ff81/cf: appending metadata at 1676307659069 (+4 ms)Flushing mob file f570113e41c274b8a0b2582dd682ff81/cf: closing flushed file at 1676307659069Flushing f570113e41c274b8a0b2582dd682ff81/cf: appending metadata at 1676307659084 (+15 ms)Flushing f570113e41c274b8a0b2582dd682ff81/cf: closing flushed file at 1676307659084Flushing f570113e41c274b8a0b2582dd682ff81/cf: reopening flushed file at 1676307659099 (+15 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for f570113e41c274b8a0b2582dd682ff81 in 39ms, sequenceid=8, compaction requested=false at 1676307659103 (+4 ms)Writing region close event to WAL at 1676307659103Running coprocessor post-close hooks at 1676307659106 (+3 ms)Closed at 1676307659106 2023-02-13 17:00:59,109 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:00:59,110 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=CLOSED 2023-02-13 17:00:59,110 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307659110"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307659110"}]},"ts":"1676307659110"} 2023-02-13 17:00:59,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=247 2023-02-13 17:00:59,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=247, state=SUCCESS; CloseRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747 in 286 msec 2023-02-13 17:00:59,111 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, UNASSIGN in 294 msec 2023-02-13 17:00:59,112 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=243 2023-02-13 17:00:59,112 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=243, state=SUCCESS; CloseRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670 in 285 msec 2023-02-13 17:00:59,113 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=243, resume processing ppid=242 2023-02-13 17:00:59,113 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, UNASSIGN in 296 msec 2023-02-13 17:00:59,114 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307659113"}]},"ts":"1676307659113"} 2023-02-13 17:00:59,114 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=DISABLED in hbase:meta 2023-02-13 17:00:59,117 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676307655966 to state=DISABLED 2023-02-13 17:00:59,118 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676307655966 in 309 msec 2023-02-13 17:00:59,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 17:00:59,565 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676307655966, procId: 242 completed 2023-02-13 17:00:59,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 } 2023-02-13 17:00:59,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676307659566 (current time:1676307659566). 2023-02-13 17:00:59,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 17:00:59,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(139): Connect 0x15f43c62 to 127.0.0.1:54083 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:00:59,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36ac4f2f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:00:59,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:00:59,576 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53712, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:00:59,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15f43c62 to 127.0.0.1:54083 2023-02-13 17:00:59,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:00:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 17:00:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 } 2023-02-13 17:00:59,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676307655966, type=EXCLUSIVE 2023-02-13 17:00:59,579 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676307655966, type=EXCLUSIVE 2023-02-13 17:00:59,580 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676307655966, type=EXCLUSIVE; timeout=600000, timestamp=1676308259580 2023-02-13 17:00:59,580 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676307655966 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676307655966 2023-02-13 17:00:59,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 } 2023-02-13 17:00:59,580 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 17:00:59,580 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 17:00:59,589 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676307655966 table=testCorruptedSnapshot-1676307655966 type=DISABLED ttl=0 } 2023-02-13 17:00:59,590 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:59,590 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 17:00:59,591 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 17:00:59,592 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c] hfiles 2023-02-13 17:00:59,592 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c 2023-02-13 17:00:59,593 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/6b5e35109ff44629bc962aa39bd36925] hfiles 2023-02-13 17:00:59,593 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/6b5e35109ff44629bc962aa39bd36925 2023-02-13 17:00:59,593 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/29756e1075774bad95cd552f57055e05] hfiles 2023-02-13 17:00:59,593 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/eb324649c13e4af88dbc2f2d549860e4] hfiles 2023-02-13 17:00:59,593 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/eb324649c13e4af88dbc2f2d549860e4 2023-02-13 17:00:59,593 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/a6e630d19d804cff8dfab678dc0b4a69] hfiles 2023-02-13 17:00:59,593 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/93866f2c0f744934afb3530f0a45adc6] hfiles 2023-02-13 17:00:59,593 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/93866f2c0f744934afb3530f0a45adc6 2023-02-13 17:00:59,593 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/a6e630d19d804cff8dfab678dc0b4a69 2023-02-13 17:00:59,593 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/29756e1075774bad95cd552f57055e05 2023-02-13 17:00:59,595 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/a87ff679a2f3e71d9181a67b7542122c20230213476fe5d0115c4dadb854b4679cc317db, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c4ca4238a0b923820dcc509a6f75849b20230213f0259ef983bd4d8db5e54997e978832c, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c81e728d9d4c2f636f067f89cc14862c202302135ce47f848d6f4a4198706abbf241a932, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/d41d8cd98f00b204e9800998ecf8427e202302130114bb04302148cf97fc307faf95443d, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/e4da3b7fbbce2345d7772b0674a318d5202302136ec2d881e3254ea59a655def7005a638, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132684611ba22b4619bbcce782b82ecb0a] hfiles 2023-02-13 17:00:59,595 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/a87ff679a2f3e71d9181a67b7542122c20230213476fe5d0115c4dadb854b4679cc317db 2023-02-13 17:00:59,595 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c4ca4238a0b923820dcc509a6f75849b20230213f0259ef983bd4d8db5e54997e978832c 2023-02-13 17:00:59,595 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c81e728d9d4c2f636f067f89cc14862c202302135ce47f848d6f4a4198706abbf241a932 2023-02-13 17:00:59,595 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/d41d8cd98f00b204e9800998ecf8427e202302130114bb04302148cf97fc307faf95443d 2023-02-13 17:00:59,595 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/e4da3b7fbbce2345d7772b0674a318d5202302136ec2d881e3254ea59a655def7005a638 2023-02-13 17:00:59,595 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132684611ba22b4619bbcce782b82ecb0a 2023-02-13 17:00:59,613 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676307655966 table=testCorruptedSnapshot-1676307655966 type=DISABLED ttl=0 } as finished. 2023-02-13 17:00:59,613 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676307655966 2023-02-13 17:00:59,614 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb0-1676307655966 2023-02-13 17:00:59,630 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb0-1676307655966 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/snaptb0-1676307655966 2023-02-13 17:00:59,646 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676307655966 of table testCorruptedSnapshot-1676307655966 completed 2023-02-13 17:00:59,646 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb0-1676307655966 2023-02-13 17:00:59,646 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.hbase-snapshot/.tmp/snaptb0-1676307655966 2023-02-13 17:00:59,646 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676307655966 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676307655966 at 1676307659580Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676307655966 table=testCorruptedSnapshot-1676307655966 type=DISABLED ttl=0 } at 1676307659589 (+9 ms)Consolidate snapshot: snaptb0-1676307655966 at 1676307659613 (+24 ms)Loading Region manifests for snaptb0-1676307655966 at 1676307659613Writing data manifest for snaptb0-1676307655966 at 1676307659616 (+3 ms)Verifying snapshot: snaptb0-1676307655966 at 1676307659624 (+8 ms)Snapshot snaptb0-1676307655966 of table testCorruptedSnapshot-1676307655966 completed at 1676307659646 (+22 ms) 2023-02-13 17:00:59,647 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676307655966, type=EXCLUSIVE 2023-02-13 17:00:59,648 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676307655966, type=EXCLUSIVE in 69 msec 2023-02-13 17:00:59,830 DEBUG [Listener at localhost/40167] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 17:00:59,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 } is done 2023-02-13 17:00:59,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676307655966 table=testCorruptedSnapshot-1676307655966 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 17:00:59,832 INFO [Listener at localhost/40167] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676307655966 2023-02-13 17:00:59,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676307655966 2023-02-13 17:00:59,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:00:59,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 17:00:59,835 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676307655966 2023-02-13 17:00:59,836 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307659836"}]},"ts":"1676307659836"} 2023-02-13 17:00:59,837 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=ENABLING in hbase:meta 2023-02-13 17:00:59,839 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 17:00:59,839 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN}] 2023-02-13 17:00:59,840 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN 2023-02-13 17:00:59,840 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN 2023-02-13 17:00:59,840 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN 2023-02-13 17:00:59,840 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN 2023-02-13 17:00:59,840 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:59,840 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:59,841 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:59,841 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=true 2023-02-13 17:00:59,841 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN 2023-02-13 17:00:59,841 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:59,841 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=true 2023-02-13 17:00:59,841 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=true 2023-02-13 17:00:59,841 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34035,1676307627747; forceNewPlan=false, retain=true 2023-02-13 17:00:59,841 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN 2023-02-13 17:00:59,841 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:59,842 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43065,1676307627716; forceNewPlan=false, retain=true 2023-02-13 17:00:59,842 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:59,842 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43327,1676307627670; forceNewPlan=false, retain=true 2023-02-13 17:00:59,991 INFO [jenkins-hbase4:40255] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 17:00:59,991 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:59,991 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:59,991 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:00:59,991 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:59,991 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:00:59,992 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307659991"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307659991"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307659991"}]},"ts":"1676307659991"} 2023-02-13 17:00:59,991 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307659991"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307659991"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307659991"}]},"ts":"1676307659991"} 2023-02-13 17:00:59,991 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307659991"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307659991"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307659991"}]},"ts":"1676307659991"} 2023-02-13 17:00:59,992 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307659991"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307659991"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307659991"}]},"ts":"1676307659991"} 2023-02-13 17:00:59,992 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307659991"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307659991"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307659991"}]},"ts":"1676307659991"} 2023-02-13 17:00:59,993 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=262, state=RUNNABLE; OpenRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:59,993 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=257, state=RUNNABLE; OpenRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:59,994 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=259, state=RUNNABLE; OpenRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:00:59,995 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=258, state=RUNNABLE; OpenRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:00:59,996 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=261, state=RUNNABLE; OpenRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:00:59,996 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:00:59,996 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307659996"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307659996"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307659996"}]},"ts":"1676307659996"} 2023-02-13 17:00:59,998 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=260, state=RUNNABLE; OpenRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:01:00,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 17:01:00,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:01:00,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:01:00,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f570113e41c274b8a0b2582dd682ff81, NAME => 'testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 17:01:00,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8d87bc00cf39a8c35d32e1f7fb7f3db5, NAME => 'testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 17:01:00,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:00,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:00,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:01:00,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:01:00,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:00,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:00,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:00,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:00,149 INFO [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:00,149 INFO [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:00,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:01:00,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 14fbb78d336eb28a243b7c7310fa32d2, NAME => 'testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 17:01:00,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:00,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:01:00,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:00,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:00,150 DEBUG [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf 2023-02-13 17:01:00,150 DEBUG [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf 2023-02-13 17:01:00,150 DEBUG [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf 2023-02-13 17:01:00,150 DEBUG [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf 2023-02-13 17:01:00,151 INFO [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f570113e41c274b8a0b2582dd682ff81 columnFamilyName cf 2023-02-13 17:01:00,151 INFO [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:00,151 INFO [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8d87bc00cf39a8c35d32e1f7fb7f3db5 columnFamilyName cf 2023-02-13 17:01:00,151 DEBUG [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf 2023-02-13 17:01:00,152 DEBUG [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf 2023-02-13 17:01:00,152 INFO [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 14fbb78d336eb28a243b7c7310fa32d2 columnFamilyName cf 2023-02-13 17:01:00,154 DEBUG [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/29756e1075774bad95cd552f57055e05 2023-02-13 17:01:00,154 DEBUG [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/eb324649c13e4af88dbc2f2d549860e4 2023-02-13 17:01:00,154 INFO [StoreOpener-8d87bc00cf39a8c35d32e1f7fb7f3db5-1] regionserver.HStore(310): Store=8d87bc00cf39a8c35d32e1f7fb7f3db5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:01:00,155 INFO [StoreOpener-f570113e41c274b8a0b2582dd682ff81-1] regionserver.HStore(310): Store=f570113e41c274b8a0b2582dd682ff81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:01:00,155 DEBUG [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/6b5e35109ff44629bc962aa39bd36925 2023-02-13 17:01:00,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:00,155 INFO [StoreOpener-14fbb78d336eb28a243b7c7310fa32d2-1] regionserver.HStore(310): Store=14fbb78d336eb28a243b7c7310fa32d2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:01:00,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:00,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:00,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:00,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:00,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:00,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:00,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:00,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:00,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f570113e41c274b8a0b2582dd682ff81; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72687338, jitterRate=0.08312574028968811}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:01:00,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f570113e41c274b8a0b2582dd682ff81: Running coprocessor pre-open hook at 1676307660149Writing region info on filesystem at 1676307660149Initializing all the Stores at 1676307660149Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307660149Cleaning up temporary data from old regions at 1676307660156 (+7 ms)Cleaning up detritus from prior splits at 1676307660157 (+1 ms)Running coprocessor post-open hooks at 1676307660159 (+2 ms)Region opened successfully at 1676307660159 2023-02-13 17:01:00,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8d87bc00cf39a8c35d32e1f7fb7f3db5; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60892090, jitterRate=-0.09263715147972107}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:01:00,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8d87bc00cf39a8c35d32e1f7fb7f3db5: Running coprocessor pre-open hook at 1676307660149Writing region info on filesystem at 1676307660149Initializing all the Stores at 1676307660149Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307660149Cleaning up temporary data from old regions at 1676307660156 (+7 ms)Cleaning up detritus from prior splits at 1676307660157 (+1 ms)Running coprocessor post-open hooks at 1676307660159 (+2 ms)Region opened successfully at 1676307660159 2023-02-13 17:01:00,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 14fbb78d336eb28a243b7c7310fa32d2; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69203498, jitterRate=0.03121247887611389}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:01:00,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 14fbb78d336eb28a243b7c7310fa32d2: Running coprocessor pre-open hook at 1676307660150Writing region info on filesystem at 1676307660150Initializing all the Stores at 1676307660150Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307660151 (+1 ms)Cleaning up temporary data from old regions at 1676307660157 (+6 ms)Cleaning up detritus from prior splits at 1676307660157Running coprocessor post-open hooks at 1676307660159 (+2 ms)Region opened successfully at 1676307660159 2023-02-13 17:01:00,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81., pid=264, masterSystemTime=1676307660145 2023-02-13 17:01:00,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5., pid=265, masterSystemTime=1676307660145 2023-02-13 17:01:00,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2., pid=268, masterSystemTime=1676307660147 2023-02-13 17:01:00,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:01:00,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:01:00,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:01:00,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d204882d59ec244668fd188df03d63c2, NAME => 'testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 17:01:00,161 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:01:00,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:00,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:01:00,162 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307660161"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307660161"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307660161"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307660161"}]},"ts":"1676307660161"} 2023-02-13 17:01:00,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:00,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:01:00,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:00,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:01:00,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:01:00,162 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:01:00,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b1c0a10356cfd3754fa8161677e0dd55, NAME => 'testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 17:01:00,162 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307660162"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307660162"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307660162"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307660162"}]},"ts":"1676307660162"} 2023-02-13 17:01:00,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:00,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:01:00,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:00,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:01:00,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:00,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:01:00,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:01:00,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 14b9e8c5d6a0e94884bc0b364ef904bf, NAME => 'testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 17:01:00,163 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:01:00,163 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307660163"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307660163"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307660163"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307660163"}]},"ts":"1676307660163"} 2023-02-13 17:01:00,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676307655966 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:00,163 INFO [StoreOpener-d204882d59ec244668fd188df03d63c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:00,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:01:00,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:00,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:00,166 DEBUG [StoreOpener-d204882d59ec244668fd188df03d63c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf 2023-02-13 17:01:00,166 DEBUG [StoreOpener-d204882d59ec244668fd188df03d63c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf 2023-02-13 17:01:00,166 INFO [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:00,166 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=257 2023-02-13 17:01:00,166 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=257, state=SUCCESS; OpenRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670 in 171 msec 2023-02-13 17:01:00,166 INFO [StoreOpener-d204882d59ec244668fd188df03d63c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d204882d59ec244668fd188df03d63c2 columnFamilyName cf 2023-02-13 17:01:00,167 DEBUG [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf 2023-02-13 17:01:00,167 DEBUG [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf 2023-02-13 17:01:00,167 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, ASSIGN in 327 msec 2023-02-13 17:01:00,167 INFO [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 14b9e8c5d6a0e94884bc0b364ef904bf columnFamilyName cf 2023-02-13 17:01:00,171 DEBUG [StoreOpener-d204882d59ec244668fd188df03d63c2-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/93866f2c0f744934afb3530f0a45adc6 2023-02-13 17:01:00,171 INFO [StoreOpener-d204882d59ec244668fd188df03d63c2-1] regionserver.HStore(310): Store=d204882d59ec244668fd188df03d63c2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:01:00,172 DEBUG [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c 2023-02-13 17:01:00,172 INFO [StoreOpener-14b9e8c5d6a0e94884bc0b364ef904bf-1] regionserver.HStore(310): Store=14b9e8c5d6a0e94884bc0b364ef904bf/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:01:00,172 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:00,173 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=259 2023-02-13 17:01:00,173 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=259, state=SUCCESS; OpenRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716 in 174 msec 2023-02-13 17:01:00,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:00,174 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=260 2023-02-13 17:01:00,174 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=260, state=SUCCESS; OpenRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747 in 171 msec 2023-02-13 17:01:00,175 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, ASSIGN in 334 msec 2023-02-13 17:01:00,175 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, ASSIGN in 335 msec 2023-02-13 17:01:00,176 INFO [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:00,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:00,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:00,177 DEBUG [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf 2023-02-13 17:01:00,177 DEBUG [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf 2023-02-13 17:01:00,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:00,177 INFO [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b1c0a10356cfd3754fa8161677e0dd55 columnFamilyName cf 2023-02-13 17:01:00,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d204882d59ec244668fd188df03d63c2; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66233866, jitterRate=-0.013038486242294312}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:01:00,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d204882d59ec244668fd188df03d63c2: Running coprocessor pre-open hook at 1676307660162Writing region info on filesystem at 1676307660162Initializing all the Stores at 1676307660163 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307660163Cleaning up temporary data from old regions at 1676307660174 (+11 ms)Cleaning up detritus from prior splits at 1676307660174Running coprocessor post-open hooks at 1676307660178 (+4 ms)Region opened successfully at 1676307660178 2023-02-13 17:01:00,178 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2., pid=266, masterSystemTime=1676307660145 2023-02-13 17:01:00,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:01:00,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:01:00,180 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:01:00,180 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307660180"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307660180"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307660180"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307660180"}]},"ts":"1676307660180"} 2023-02-13 17:01:00,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:00,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 14b9e8c5d6a0e94884bc0b364ef904bf; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71793170, jitterRate=0.06980159878730774}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:01:00,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 14b9e8c5d6a0e94884bc0b364ef904bf: Running coprocessor pre-open hook at 1676307660164Writing region info on filesystem at 1676307660164Initializing all the Stores at 1676307660165 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307660165Cleaning up temporary data from old regions at 1676307660178 (+13 ms)Cleaning up detritus from prior splits at 1676307660178Running coprocessor post-open hooks at 1676307660181 (+3 ms)Region opened successfully at 1676307660181 2023-02-13 17:01:00,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf., pid=267, masterSystemTime=1676307660147 2023-02-13 17:01:00,182 DEBUG [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] regionserver.HStore(539): loaded hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/a6e630d19d804cff8dfab678dc0b4a69 2023-02-13 17:01:00,182 INFO [StoreOpener-b1c0a10356cfd3754fa8161677e0dd55-1] regionserver.HStore(310): Store=b1c0a10356cfd3754fa8161677e0dd55/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:01:00,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:00,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:01:00,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:01:00,183 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=258 2023-02-13 17:01:00,183 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=258, state=SUCCESS; OpenRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670 in 186 msec 2023-02-13 17:01:00,184 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:01:00,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:00,184 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307660184"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307660184"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307660184"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307660184"}]},"ts":"1676307660184"} 2023-02-13 17:01:00,184 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, ASSIGN in 344 msec 2023-02-13 17:01:00,186 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=261 2023-02-13 17:01:00,186 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=261, state=SUCCESS; OpenRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747 in 189 msec 2023-02-13 17:01:00,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:00,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b1c0a10356cfd3754fa8161677e0dd55; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70436241, jitterRate=0.04958178102970123}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:01:00,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b1c0a10356cfd3754fa8161677e0dd55: Running coprocessor pre-open hook at 1676307660163Writing region info on filesystem at 1676307660163Initializing all the Stores at 1676307660172 (+9 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676307660172Cleaning up temporary data from old regions at 1676307660184 (+12 ms)Cleaning up detritus from prior splits at 1676307660185 (+1 ms)Running coprocessor post-open hooks at 1676307660187 (+2 ms)Region opened successfully at 1676307660187 2023-02-13 17:01:00,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, ASSIGN in 347 msec 2023-02-13 17:01:00,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55., pid=263, masterSystemTime=1676307660145 2023-02-13 17:01:00,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:01:00,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:01:00,189 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:01:00,189 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307660189"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307660189"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307660189"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307660189"}]},"ts":"1676307660189"} 2023-02-13 17:01:00,191 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=262 2023-02-13 17:01:00,191 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=262, state=SUCCESS; OpenRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716 in 197 msec 2023-02-13 17:01:00,192 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=262, resume processing ppid=256 2023-02-13 17:01:00,192 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, ASSIGN in 352 msec 2023-02-13 17:01:00,193 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307660192"}]},"ts":"1676307660192"} 2023-02-13 17:01:00,194 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=ENABLED in hbase:meta 2023-02-13 17:01:00,195 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676307655966' was successfully enabled. 2023-02-13 17:01:00,197 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676307655966 in 364 msec 2023-02-13 17:01:00,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 17:01:00,587 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676307655966, procId: 256 completed 2023-02-13 17:01:00,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:01:00,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34035] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:01:00,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:01:00,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43327] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:01:00,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:01:00,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43065] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 17:01:00,599 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676307655966 2023-02-13 17:01:00,599 DEBUG [Listener at localhost/40167] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:01:00,599 INFO [Listener at localhost/40167] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:01:00,704 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 17:01:00,763 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 17:01:00,853 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 17:01:00,913 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 17:01:00,976 DEBUG [Listener at localhost/40167] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 17:01:01,172 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x594888d9 to 127.0.0.1:54083 2023-02-13 17:01:01,172 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:01:01,772 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: be910e1657a440b58fb8e57ec0573d5c in the real (hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c) or archive (hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c) directory for the primary table. 2023-02-13 17:01:01,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: be910e1657a440b58fb8e57ec0573d5c in the real (hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c) or archive (hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:01:01,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] ipc.CallRunner(144): callId: 364 service: MasterService methodName: RestoreSnapshot size: 115 connection: 172.31.14.131:33796 deadline: 1676307721768, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: be910e1657a440b58fb8e57ec0573d5c in the real (hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c) or archive (hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/be910e1657a440b58fb8e57ec0573d5c) directory for the primary table. 2023-02-13 17:01:01,777 INFO [Listener at localhost/40167] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676307655966 2023-02-13 17:01:01,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676307655966 2023-02-13 17:01:01,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:01:01,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 17:01:01,781 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307661781"}]},"ts":"1676307661781"} 2023-02-13 17:01:01,782 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=DISABLING in hbase:meta 2023-02-13 17:01:01,784 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676307655966 to state=DISABLING 2023-02-13 17:01:01,785 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, UNASSIGN}] 2023-02-13 17:01:01,785 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, UNASSIGN 2023-02-13 17:01:01,785 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, UNASSIGN 2023-02-13 17:01:01,786 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, UNASSIGN 2023-02-13 17:01:01,786 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, UNASSIGN 2023-02-13 17:01:01,786 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, UNASSIGN 2023-02-13 17:01:01,786 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:01:01,786 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307661786"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307661786"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307661786"}]},"ts":"1676307661786"} 2023-02-13 17:01:01,786 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:01:01,786 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:01:01,786 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307661786"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307661786"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307661786"}]},"ts":"1676307661786"} 2023-02-13 17:01:01,786 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307661786"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307661786"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307661786"}]},"ts":"1676307661786"} 2023-02-13 17:01:01,786 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:01:01,786 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:01:01,787 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307661786"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307661786"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307661786"}]},"ts":"1676307661786"} 2023-02-13 17:01:01,787 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307661786"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307661786"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307661786"}]},"ts":"1676307661786"} 2023-02-13 17:01:01,787 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=275, state=RUNNABLE; CloseRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:01:01,788 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=273, state=RUNNABLE; CloseRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:01:01,788 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=272, state=RUNNABLE; CloseRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:01:01,789 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=274, state=RUNNABLE; CloseRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747}] 2023-02-13 17:01:01,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=276, state=RUNNABLE; CloseRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716}] 2023-02-13 17:01:01,790 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, UNASSIGN 2023-02-13 17:01:01,791 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:01:01,791 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307661791"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307661791"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307661791"}]},"ts":"1676307661791"} 2023-02-13 17:01:01,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670}] 2023-02-13 17:01:01,939 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:01,939 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:01,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b1c0a10356cfd3754fa8161677e0dd55, disabling compactions & flushes 2023-02-13 17:01:01,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 14b9e8c5d6a0e94884bc0b364ef904bf, disabling compactions & flushes 2023-02-13 17:01:01,939 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:01:01,939 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:01:01,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:01:01,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:01:01,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. after waiting 0 ms 2023-02-13 17:01:01,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:01:01,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. after waiting 0 ms 2023-02-13 17:01:01,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:01:01,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 14b9e8c5d6a0e94884bc0b364ef904bf 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-13 17:01:01,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing b1c0a10356cfd3754fa8161677e0dd55 1/1 column families, dataSize=22.05 KB heapSize=47.64 KB 2023-02-13 17:01:01,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:01,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f570113e41c274b8a0b2582dd682ff81, disabling compactions & flushes 2023-02-13 17:01:01,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:01:01,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:01:01,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. after waiting 0 ms 2023-02-13 17:01:01,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:01:01,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f570113e41c274b8a0b2582dd682ff81 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-13 17:01:01,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230213fa7da7483e1b446e81a16796cabfc564 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/d41d8cd98f00b204e9800998ecf8427e20230213fa7da7483e1b446e81a16796cabfc564 2023-02-13 17:01:01,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230213d0961a44cae9463b86ff736afd1bed61 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/e4da3b7fbbce2345d7772b0674a318d520230213d0961a44cae9463b86ff736afd1bed61 2023-02-13 17:01:01,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213471c8b6d5a7a49b79801dbb3f6296699 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/a87ff679a2f3e71d9181a67b7542122c20230213471c8b6d5a7a49b79801dbb3f6296699 2023-02-13 17:01:01,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/.tmp/cf/828bcff35f5441149de31c93296b132a 2023-02-13 17:01:01,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/.tmp/cf/88846ebc6f634e83993679dde344e477 2023-02-13 17:01:01,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/.tmp/cf/fe141f908018400b96b6f9ad5f7bb4c5 2023-02-13 17:01:01,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/.tmp/cf/828bcff35f5441149de31c93296b132a as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/828bcff35f5441149de31c93296b132a 2023-02-13 17:01:01,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/.tmp/cf/88846ebc6f634e83993679dde344e477 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/88846ebc6f634e83993679dde344e477 2023-02-13 17:01:01,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/.tmp/cf/fe141f908018400b96b6f9ad5f7bb4c5 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/fe141f908018400b96b6f9ad5f7bb4c5 2023-02-13 17:01:01,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/828bcff35f5441149de31c93296b132a, entries=33, sequenceid=15, filesize=10.6 K 2023-02-13 17:01:01,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for f570113e41c274b8a0b2582dd682ff81 in 46ms, sequenceid=15, compaction requested=false 2023-02-13 17:01:01,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/88846ebc6f634e83993679dde344e477, entries=337, sequenceid=15, filesize=63.0 K 2023-02-13 17:01:01,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.05 KB/22575, heapSize ~47.63 KB/48768, currentSize=0 B/0 for b1c0a10356cfd3754fa8161677e0dd55 in 49ms, sequenceid=15, compaction requested=false 2023-02-13 17:01:01,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/fe141f908018400b96b6f9ad5f7bb4c5, entries=35, sequenceid=15, filesize=10.9 K 2023-02-13 17:01:01,991 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 14b9e8c5d6a0e94884bc0b364ef904bf in 51ms, sequenceid=15, compaction requested=false 2023-02-13 17:01:01,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:01:01,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. 2023-02-13 17:01:01,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f570113e41c274b8a0b2582dd682ff81: Waiting for close lock at 1676307661940Running coprocessor pre-close hooks at 1676307661940Disabling compacts and flushes for region at 1676307661940Disabling writes for close at 1676307661940Obtaining lock to block concurrent updates at 1676307661941 (+1 ms)Preparing flush snapshotting stores in f570113e41c274b8a0b2582dd682ff81 at 1676307661941Finished memstore snapshotting testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676307661942 (+1 ms)Flushing stores of testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81. at 1676307661942Flushing f570113e41c274b8a0b2582dd682ff81/cf: creating writer at 1676307661943 (+1 ms)Flushing mob file f570113e41c274b8a0b2582dd682ff81/cf: appending metadata at 1676307661946 (+3 ms)Flushing mob file f570113e41c274b8a0b2582dd682ff81/cf: closing flushed file at 1676307661946Flushing f570113e41c274b8a0b2582dd682ff81/cf: appending metadata at 1676307661968 (+22 ms)Flushing f570113e41c274b8a0b2582dd682ff81/cf: closing flushed file at 1676307661968Flushing f570113e41c274b8a0b2582dd682ff81/cf: reopening flushed file at 1676307661982 (+14 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for f570113e41c274b8a0b2582dd682ff81 in 46ms, sequenceid=15, compaction requested=false at 1676307661986 (+4 ms)Writing region close event to WAL at 1676307661986Running coprocessor post-close hooks at 1676307661992 (+6 ms)Closed at 1676307661992 2023-02-13 17:01:01,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:01:01,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. 2023-02-13 17:01:01,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b1c0a10356cfd3754fa8161677e0dd55: Waiting for close lock at 1676307661939Running coprocessor pre-close hooks at 1676307661939Disabling compacts and flushes for region at 1676307661939Disabling writes for close at 1676307661940 (+1 ms)Obtaining lock to block concurrent updates at 1676307661940Preparing flush snapshotting stores in b1c0a10356cfd3754fa8161677e0dd55 at 1676307661940Finished memstore snapshotting testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55., syncing WAL and waiting on mvcc, flushsize=dataSize=22575, getHeapSize=48768, getOffHeapSize=0, getCellsCount=337 at 1676307661940Flushing stores of testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55. at 1676307661940Flushing b1c0a10356cfd3754fa8161677e0dd55/cf: creating writer at 1676307661941 (+1 ms)Flushing mob file b1c0a10356cfd3754fa8161677e0dd55/cf: appending metadata at 1676307661947 (+6 ms)Flushing mob file b1c0a10356cfd3754fa8161677e0dd55/cf: closing flushed file at 1676307661947Flushing b1c0a10356cfd3754fa8161677e0dd55/cf: appending metadata at 1676307661968 (+21 ms)Flushing b1c0a10356cfd3754fa8161677e0dd55/cf: closing flushed file at 1676307661968Flushing b1c0a10356cfd3754fa8161677e0dd55/cf: reopening flushed file at 1676307661984 (+16 ms)Finished flush of dataSize ~22.05 KB/22575, heapSize ~47.63 KB/48768, currentSize=0 B/0 for b1c0a10356cfd3754fa8161677e0dd55 in 49ms, sequenceid=15, compaction requested=false at 1676307661990 (+6 ms)Writing region close event to WAL at 1676307661990Running coprocessor post-close hooks at 1676307661993 (+3 ms)Closed at 1676307661993 2023-02-13 17:01:01,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:01:01,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. 2023-02-13 17:01:01,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 14b9e8c5d6a0e94884bc0b364ef904bf: Waiting for close lock at 1676307661939Running coprocessor pre-close hooks at 1676307661939Disabling compacts and flushes for region at 1676307661939Disabling writes for close at 1676307661939Obtaining lock to block concurrent updates at 1676307661940 (+1 ms)Preparing flush snapshotting stores in 14b9e8c5d6a0e94884bc0b364ef904bf at 1676307661940Finished memstore snapshotting testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf., syncing WAL and waiting on mvcc, flushsize=dataSize=2343, getHeapSize=5280, getOffHeapSize=0, getCellsCount=35 at 1676307661940Flushing stores of testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf. at 1676307661940Flushing 14b9e8c5d6a0e94884bc0b364ef904bf/cf: creating writer at 1676307661941 (+1 ms)Flushing mob file 14b9e8c5d6a0e94884bc0b364ef904bf/cf: appending metadata at 1676307661946 (+5 ms)Flushing mob file 14b9e8c5d6a0e94884bc0b364ef904bf/cf: closing flushed file at 1676307661946Flushing 14b9e8c5d6a0e94884bc0b364ef904bf/cf: appending metadata at 1676307661968 (+22 ms)Flushing 14b9e8c5d6a0e94884bc0b364ef904bf/cf: closing flushed file at 1676307661968Flushing 14b9e8c5d6a0e94884bc0b364ef904bf/cf: reopening flushed file at 1676307661985 (+17 ms)Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 14b9e8c5d6a0e94884bc0b364ef904bf in 51ms, sequenceid=15, compaction requested=false at 1676307661991 (+6 ms)Writing region close event to WAL at 1676307661991Running coprocessor post-close hooks at 1676307661994 (+3 ms)Closed at 1676307661994 2023-02-13 17:01:01,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:01,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:01,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d204882d59ec244668fd188df03d63c2, disabling compactions & flushes 2023-02-13 17:01:01,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:01:01,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:01:01,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. after waiting 0 ms 2023-02-13 17:01:01,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:01:01,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d204882d59ec244668fd188df03d63c2 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-13 17:01:01,995 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=f570113e41c274b8a0b2582dd682ff81, regionState=CLOSED 2023-02-13 17:01:01,995 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307661995"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307661995"}]},"ts":"1676307661995"} 2023-02-13 17:01:01,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:01,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:01,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8d87bc00cf39a8c35d32e1f7fb7f3db5, disabling compactions & flushes 2023-02-13 17:01:01,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:01:01,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:01:01,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. after waiting 0 ms 2023-02-13 17:01:01,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:01:01,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 8d87bc00cf39a8c35d32e1f7fb7f3db5 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-13 17:01:01,998 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=b1c0a10356cfd3754fa8161677e0dd55, regionState=CLOSED 2023-02-13 17:01:01,998 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676307661998"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307661998"}]},"ts":"1676307661998"} 2023-02-13 17:01:01,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:01,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:01,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 14fbb78d336eb28a243b7c7310fa32d2, disabling compactions & flushes 2023-02-13 17:01:01,999 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=14b9e8c5d6a0e94884bc0b364ef904bf, regionState=CLOSED 2023-02-13 17:01:01,999 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307661999"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307661999"}]},"ts":"1676307661999"} 2023-02-13 17:01:01,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:01:01,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:01:01,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. after waiting 0 ms 2023-02-13 17:01:01,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:01:01,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 14fbb78d336eb28a243b7c7310fa32d2 1/1 column families, dataSize=1.57 KB heapSize=3.63 KB 2023-02-13 17:01:02,000 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-02-13 17:01:02,000 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure f570113e41c274b8a0b2582dd682ff81, server=jenkins-hbase4.apache.org,43327,1676307627670 in 206 msec 2023-02-13 17:01:02,004 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=f570113e41c274b8a0b2582dd682ff81, UNASSIGN in 215 msec 2023-02-13 17:01:02,004 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=276 2023-02-13 17:01:02,004 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=276, state=SUCCESS; CloseRegionProcedure b1c0a10356cfd3754fa8161677e0dd55, server=jenkins-hbase4.apache.org,43065,1676307627716 in 209 msec 2023-02-13 17:01:02,005 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=275 2023-02-13 17:01:02,005 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=275, state=SUCCESS; CloseRegionProcedure 14b9e8c5d6a0e94884bc0b364ef904bf, server=jenkins-hbase4.apache.org,34035,1676307627747 in 214 msec 2023-02-13 17:01:02,005 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=b1c0a10356cfd3754fa8161677e0dd55, UNASSIGN in 219 msec 2023-02-13 17:01:02,006 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14b9e8c5d6a0e94884bc0b364ef904bf, UNASSIGN in 220 msec 2023-02-13 17:01:02,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302137ed531bb946641308894efe0023a3d32 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c81e728d9d4c2f636f067f89cc14862c202302137ed531bb946641308894efe0023a3d32 2023-02-13 17:01:02,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021307921c99a3ef41b9bb4f912255c04044 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c4ca4238a0b923820dcc509a6f75849b2023021307921c99a3ef41b9bb4f912255c04044 2023-02-13 17:01:02,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213b413fe6ac48a44659c0debe530144961 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213b413fe6ac48a44659c0debe530144961 2023-02-13 17:01:02,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 17:01:02,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/.tmp/cf/95a17fd538604f59947289064bf9cabd 2023-02-13 17:01:02,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/.tmp/cf/f535dfd0338340eba4670c1320b330bf 2023-02-13 17:01:02,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/.tmp/cf/95a17fd538604f59947289064bf9cabd as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/95a17fd538604f59947289064bf9cabd 2023-02-13 17:01:02,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/.tmp/cf/128fac89fce242f0a0a29168aab732df 2023-02-13 17:01:02,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/.tmp/cf/f535dfd0338340eba4670c1320b330bf as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/f535dfd0338340eba4670c1320b330bf 2023-02-13 17:01:02,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/95a17fd538604f59947289064bf9cabd, entries=37, sequenceid=15, filesize=11.3 K 2023-02-13 17:01:02,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/.tmp/cf/128fac89fce242f0a0a29168aab732df as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/128fac89fce242f0a0a29168aab732df 2023-02-13 17:01:02,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/f535dfd0338340eba4670c1320b330bf, entries=24, sequenceid=15, filesize=9.0 K 2023-02-13 17:01:02,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 8d87bc00cf39a8c35d32e1f7fb7f3db5 in 44ms, sequenceid=15, compaction requested=false 2023-02-13 17:01:02,041 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 14fbb78d336eb28a243b7c7310fa32d2 in 42ms, sequenceid=15, compaction requested=false 2023-02-13 17:01:02,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/128fac89fce242f0a0a29168aab732df, entries=34, sequenceid=15, filesize=10.8 K 2023-02-13 17:01:02,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for d204882d59ec244668fd188df03d63c2 in 48ms, sequenceid=15, compaction requested=false 2023-02-13 17:01:02,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:01:02,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:01:02,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. 2023-02-13 17:01:02,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8d87bc00cf39a8c35d32e1f7fb7f3db5: Waiting for close lock at 1676307661996Running coprocessor pre-close hooks at 1676307661996Disabling compacts and flushes for region at 1676307661996Disabling writes for close at 1676307661996Obtaining lock to block concurrent updates at 1676307661996Preparing flush snapshotting stores in 8d87bc00cf39a8c35d32e1f7fb7f3db5 at 1676307661996Finished memstore snapshotting testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5., syncing WAL and waiting on mvcc, flushsize=dataSize=2477, getHeapSize=5568, getOffHeapSize=0, getCellsCount=37 at 1676307661997 (+1 ms)Flushing stores of testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5. at 1676307661997Flushing 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: creating writer at 1676307661998 (+1 ms)Flushing mob file 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: appending metadata at 1676307662004 (+6 ms)Flushing mob file 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: closing flushed file at 1676307662004Flushing 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: appending metadata at 1676307662023 (+19 ms)Flushing 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: closing flushed file at 1676307662023Flushing 8d87bc00cf39a8c35d32e1f7fb7f3db5/cf: reopening flushed file at 1676307662037 (+14 ms)Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 8d87bc00cf39a8c35d32e1f7fb7f3db5 in 44ms, sequenceid=15, compaction requested=false at 1676307662040 (+3 ms)Writing region close event to WAL at 1676307662041 (+1 ms)Running coprocessor post-close hooks at 1676307662044 (+3 ms)Closed at 1676307662044 2023-02-13 17:01:02,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. 2023-02-13 17:01:02,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 14fbb78d336eb28a243b7c7310fa32d2: Waiting for close lock at 1676307661999Running coprocessor pre-close hooks at 1676307661999Disabling compacts and flushes for region at 1676307661999Disabling writes for close at 1676307661999Obtaining lock to block concurrent updates at 1676307661999Preparing flush snapshotting stores in 14fbb78d336eb28a243b7c7310fa32d2 at 1676307661999Finished memstore snapshotting testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2., syncing WAL and waiting on mvcc, flushsize=dataSize=1606, getHeapSize=3696, getOffHeapSize=0, getCellsCount=24 at 1676307662000 (+1 ms)Flushing stores of testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2. at 1676307662000Flushing 14fbb78d336eb28a243b7c7310fa32d2/cf: creating writer at 1676307662001 (+1 ms)Flushing mob file 14fbb78d336eb28a243b7c7310fa32d2/cf: appending metadata at 1676307662008 (+7 ms)Flushing mob file 14fbb78d336eb28a243b7c7310fa32d2/cf: closing flushed file at 1676307662008Flushing 14fbb78d336eb28a243b7c7310fa32d2/cf: appending metadata at 1676307662023 (+15 ms)Flushing 14fbb78d336eb28a243b7c7310fa32d2/cf: closing flushed file at 1676307662023Flushing 14fbb78d336eb28a243b7c7310fa32d2/cf: reopening flushed file at 1676307662038 (+15 ms)Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 14fbb78d336eb28a243b7c7310fa32d2 in 42ms, sequenceid=15, compaction requested=false at 1676307662041 (+3 ms)Writing region close event to WAL at 1676307662042 (+1 ms)Running coprocessor post-close hooks at 1676307662044 (+2 ms)Closed at 1676307662044 2023-02-13 17:01:02,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:02,046 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=8d87bc00cf39a8c35d32e1f7fb7f3db5, regionState=CLOSED 2023-02-13 17:01:02,046 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307662046"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307662046"}]},"ts":"1676307662046"} 2023-02-13 17:01:02,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:02,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 17:01:02,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. 2023-02-13 17:01:02,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d204882d59ec244668fd188df03d63c2: Waiting for close lock at 1676307661994Running coprocessor pre-close hooks at 1676307661994Disabling compacts and flushes for region at 1676307661994Disabling writes for close at 1676307661995 (+1 ms)Obtaining lock to block concurrent updates at 1676307661995Preparing flush snapshotting stores in d204882d59ec244668fd188df03d63c2 at 1676307661995Finished memstore snapshotting testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676307661995Flushing stores of testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2. at 1676307661995Flushing d204882d59ec244668fd188df03d63c2/cf: creating writer at 1676307661996 (+1 ms)Flushing mob file d204882d59ec244668fd188df03d63c2/cf: appending metadata at 1676307662008 (+12 ms)Flushing mob file d204882d59ec244668fd188df03d63c2/cf: closing flushed file at 1676307662008Flushing d204882d59ec244668fd188df03d63c2/cf: appending metadata at 1676307662023 (+15 ms)Flushing d204882d59ec244668fd188df03d63c2/cf: closing flushed file at 1676307662023Flushing d204882d59ec244668fd188df03d63c2/cf: reopening flushed file at 1676307662040 (+17 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for d204882d59ec244668fd188df03d63c2 in 48ms, sequenceid=15, compaction requested=false at 1676307662043 (+3 ms)Writing region close event to WAL at 1676307662044 (+1 ms)Running coprocessor post-close hooks at 1676307662047 (+3 ms)Closed at 1676307662047 2023-02-13 17:01:02,048 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=14fbb78d336eb28a243b7c7310fa32d2, regionState=CLOSED 2023-02-13 17:01:02,048 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307662048"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307662048"}]},"ts":"1676307662048"} 2023-02-13 17:01:02,048 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:02,048 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=d204882d59ec244668fd188df03d63c2, regionState=CLOSED 2023-02-13 17:01:02,048 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676307662048"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307662048"}]},"ts":"1676307662048"} 2023-02-13 17:01:02,049 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=273 2023-02-13 17:01:02,049 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=273, state=SUCCESS; CloseRegionProcedure 8d87bc00cf39a8c35d32e1f7fb7f3db5, server=jenkins-hbase4.apache.org,43065,1676307627716 in 259 msec 2023-02-13 17:01:02,050 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=8d87bc00cf39a8c35d32e1f7fb7f3db5, UNASSIGN in 264 msec 2023-02-13 17:01:02,050 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=274 2023-02-13 17:01:02,050 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=274, state=SUCCESS; CloseRegionProcedure 14fbb78d336eb28a243b7c7310fa32d2, server=jenkins-hbase4.apache.org,34035,1676307627747 in 260 msec 2023-02-13 17:01:02,051 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=272 2023-02-13 17:01:02,051 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=272, state=SUCCESS; CloseRegionProcedure d204882d59ec244668fd188df03d63c2, server=jenkins-hbase4.apache.org,43327,1676307627670 in 261 msec 2023-02-13 17:01:02,051 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=14fbb78d336eb28a243b7c7310fa32d2, UNASSIGN in 265 msec 2023-02-13 17:01:02,052 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=272, resume processing ppid=270 2023-02-13 17:01:02,052 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676307655966, region=d204882d59ec244668fd188df03d63c2, UNASSIGN in 266 msec 2023-02-13 17:01:02,052 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307662052"}]},"ts":"1676307662052"} 2023-02-13 17:01:02,053 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676307655966, state=DISABLED in hbase:meta 2023-02-13 17:01:02,055 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676307655966 to state=DISABLED 2023-02-13 17:01:02,056 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676307655966 in 277 msec 2023-02-13 17:01:02,260 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 17:01:02,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 17:01:02,533 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676307655966, procId: 270 completed 2023-02-13 17:01:02,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1676307655966 2023-02-13 17:01:02,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:01:02,536 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:01:02,537 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:01:02,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-13 17:01:02,543 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:02,543 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:02,543 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:02,543 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:02,543 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:02,543 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:02,545 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/recovered.edits] 2023-02-13 17:01:02,545 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/recovered.edits] 2023-02-13 17:01:02,545 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/recovered.edits] 2023-02-13 17:01:02,545 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/recovered.edits] 2023-02-13 17:01:02,545 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/recovered.edits] 2023-02-13 17:01:02,545 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf, FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/recovered.edits] 2023-02-13 17:01:02,549 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/6b5e35109ff44629bc962aa39bd36925 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/6b5e35109ff44629bc962aa39bd36925 2023-02-13 17:01:02,550 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/828bcff35f5441149de31c93296b132a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/828bcff35f5441149de31c93296b132a 2023-02-13 17:01:02,550 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/128fac89fce242f0a0a29168aab732df to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/128fac89fce242f0a0a29168aab732df 2023-02-13 17:01:02,550 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/88846ebc6f634e83993679dde344e477 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/88846ebc6f634e83993679dde344e477 2023-02-13 17:01:02,550 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/29756e1075774bad95cd552f57055e05 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/29756e1075774bad95cd552f57055e05 2023-02-13 17:01:02,550 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/fe141f908018400b96b6f9ad5f7bb4c5 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/cf/fe141f908018400b96b6f9ad5f7bb4c5 2023-02-13 17:01:02,551 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/f535dfd0338340eba4670c1320b330bf to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/cf/f535dfd0338340eba4670c1320b330bf 2023-02-13 17:01:02,551 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/eb324649c13e4af88dbc2f2d549860e4 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/cf/eb324649c13e4af88dbc2f2d549860e4 2023-02-13 17:01:02,552 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/93866f2c0f744934afb3530f0a45adc6 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/cf/93866f2c0f744934afb3530f0a45adc6 2023-02-13 17:01:02,552 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/a6e630d19d804cff8dfab678dc0b4a69 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/cf/a6e630d19d804cff8dfab678dc0b4a69 2023-02-13 17:01:02,552 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/95a17fd538604f59947289064bf9cabd to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/cf/95a17fd538604f59947289064bf9cabd 2023-02-13 17:01:02,554 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf/recovered.edits/18.seqid 2023-02-13 17:01:02,555 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14b9e8c5d6a0e94884bc0b364ef904bf 2023-02-13 17:01:02,555 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2/recovered.edits/18.seqid 2023-02-13 17:01:02,556 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55/recovered.edits/18.seqid 2023-02-13 17:01:02,556 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81/recovered.edits/18.seqid 2023-02-13 17:01:02,556 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2/recovered.edits/18.seqid 2023-02-13 17:01:02,556 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/recovered.edits/18.seqid to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5/recovered.edits/18.seqid 2023-02-13 17:01:02,557 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/14fbb78d336eb28a243b7c7310fa32d2 2023-02-13 17:01:02,557 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/b1c0a10356cfd3754fa8161677e0dd55 2023-02-13 17:01:02,557 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/f570113e41c274b8a0b2582dd682ff81 2023-02-13 17:01:02,557 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/8d87bc00cf39a8c35d32e1f7fb7f3db5 2023-02-13 17:01:02,557 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/.tmp/data/default/testCorruptedSnapshot-1676307655966/d204882d59ec244668fd188df03d63c2 2023-02-13 17:01:02,557 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676307655966 regions 2023-02-13 17:01:02,558 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237 2023-02-13 17:01:02,558 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf] 2023-02-13 17:01:02,561 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/a87ff679a2f3e71d9181a67b7542122c20230213471c8b6d5a7a49b79801dbb3f6296699 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/a87ff679a2f3e71d9181a67b7542122c20230213471c8b6d5a7a49b79801dbb3f6296699 2023-02-13 17:01:02,562 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/a87ff679a2f3e71d9181a67b7542122c20230213476fe5d0115c4dadb854b4679cc317db to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/a87ff679a2f3e71d9181a67b7542122c20230213476fe5d0115c4dadb854b4679cc317db 2023-02-13 17:01:02,564 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c4ca4238a0b923820dcc509a6f75849b2023021307921c99a3ef41b9bb4f912255c04044 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c4ca4238a0b923820dcc509a6f75849b2023021307921c99a3ef41b9bb4f912255c04044 2023-02-13 17:01:02,565 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c4ca4238a0b923820dcc509a6f75849b20230213f0259ef983bd4d8db5e54997e978832c to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c4ca4238a0b923820dcc509a6f75849b20230213f0259ef983bd4d8db5e54997e978832c 2023-02-13 17:01:02,566 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c81e728d9d4c2f636f067f89cc14862c202302135ce47f848d6f4a4198706abbf241a932 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c81e728d9d4c2f636f067f89cc14862c202302135ce47f848d6f4a4198706abbf241a932 2023-02-13 17:01:02,567 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c81e728d9d4c2f636f067f89cc14862c202302137ed531bb946641308894efe0023a3d32 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/c81e728d9d4c2f636f067f89cc14862c202302137ed531bb946641308894efe0023a3d32 2023-02-13 17:01:02,567 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/d41d8cd98f00b204e9800998ecf8427e202302130114bb04302148cf97fc307faf95443d to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/d41d8cd98f00b204e9800998ecf8427e202302130114bb04302148cf97fc307faf95443d 2023-02-13 17:01:02,568 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/d41d8cd98f00b204e9800998ecf8427e20230213fa7da7483e1b446e81a16796cabfc564 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/d41d8cd98f00b204e9800998ecf8427e20230213fa7da7483e1b446e81a16796cabfc564 2023-02-13 17:01:02,569 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/e4da3b7fbbce2345d7772b0674a318d5202302136ec2d881e3254ea59a655def7005a638 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/e4da3b7fbbce2345d7772b0674a318d5202302136ec2d881e3254ea59a655def7005a638 2023-02-13 17:01:02,570 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/e4da3b7fbbce2345d7772b0674a318d520230213d0961a44cae9463b86ff736afd1bed61 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/e4da3b7fbbce2345d7772b0674a318d520230213d0961a44cae9463b86ff736afd1bed61 2023-02-13 17:01:02,571 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132684611ba22b4619bbcce782b82ecb0a to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132684611ba22b4619bbcce782b82ecb0a 2023-02-13 17:01:02,572 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213b413fe6ac48a44659c0debe530144961 to hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/archive/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213b413fe6ac48a44659c0debe530144961 2023-02-13 17:01:02,572 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/mobdir/data/default/testCorruptedSnapshot-1676307655966/9766f6313d81f85bb642e8251627b237 2023-02-13 17:01:02,573 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:01:02,575 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676307655966 from hbase:meta 2023-02-13 17:01:02,577 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676307655966' descriptor. 2023-02-13 17:01:02,578 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:01:02,578 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676307655966' from region states. 2023-02-13 17:01:02,578 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307662578"}]},"ts":"9223372036854775807"} 2023-02-13 17:01:02,578 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307662578"}]},"ts":"9223372036854775807"} 2023-02-13 17:01:02,578 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307662578"}]},"ts":"9223372036854775807"} 2023-02-13 17:01:02,578 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307662578"}]},"ts":"9223372036854775807"} 2023-02-13 17:01:02,578 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307662578"}]},"ts":"9223372036854775807"} 2023-02-13 17:01:02,578 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676307662578"}]},"ts":"9223372036854775807"} 2023-02-13 17:01:02,580 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 17:01:02,580 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f570113e41c274b8a0b2582dd682ff81, NAME => 'testCorruptedSnapshot-1676307655966,,1676307655968.f570113e41c274b8a0b2582dd682ff81.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => d204882d59ec244668fd188df03d63c2, NAME => 'testCorruptedSnapshot-1676307655966,1,1676307655968.d204882d59ec244668fd188df03d63c2.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 8d87bc00cf39a8c35d32e1f7fb7f3db5, NAME => 'testCorruptedSnapshot-1676307655966,2,1676307655968.8d87bc00cf39a8c35d32e1f7fb7f3db5.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 14fbb78d336eb28a243b7c7310fa32d2, NAME => 'testCorruptedSnapshot-1676307655966,3,1676307655968.14fbb78d336eb28a243b7c7310fa32d2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 14b9e8c5d6a0e94884bc0b364ef904bf, NAME => 'testCorruptedSnapshot-1676307655966,4,1676307655968.14b9e8c5d6a0e94884bc0b364ef904bf.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b1c0a10356cfd3754fa8161677e0dd55, NAME => 'testCorruptedSnapshot-1676307655966,5,1676307655968.b1c0a10356cfd3754fa8161677e0dd55.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 17:01:02,580 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676307655966' as deleted. 2023-02-13 17:01:02,580 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676307655966","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676307662580"}]},"ts":"9223372036854775807"} 2023-02-13 17:01:02,581 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676307655966 state from META 2023-02-13 17:01:02,583 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676307655966 2023-02-13 17:01:02,583 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676307655966 in 49 msec 2023-02-13 17:01:02,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-13 17:01:02,788 INFO [Listener at localhost/40167] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676307655966, procId: 283 completed 2023-02-13 17:01:02,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676307655966" 2023-02-13 17:01:02,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676307655966 2023-02-13 17:01:02,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676307655966" 2023-02-13 17:01:02,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40255] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676307655966 2023-02-13 17:01:02,813 INFO [Listener at localhost/40167] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=418 (was 422), OpenFileDescriptor=803 (was 809), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=293 (was 299), ProcessCount=168 (was 168), AvailableMemoryMB=851 (was 879) 2023-02-13 17:01:02,813 INFO [Listener at localhost/40167] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 17:01:02,813 INFO [Listener at localhost/40167] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 17:01:02,814 DEBUG [Listener at localhost/40167] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c633018 to 127.0.0.1:54083 2023-02-13 17:01:02,814 DEBUG [Listener at localhost/40167] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:01:02,814 DEBUG [Listener at localhost/40167] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 17:01:02,814 DEBUG [Listener at localhost/40167] util.JVMClusterUtil(257): Found active master hash=1483463506, stopped=false 2023-02-13 17:01:02,815 INFO [Listener at localhost/40167] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,40255,1676307626519 2023-02-13 17:01:02,817 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:01:02,817 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:01:02,817 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:01:02,817 INFO [Listener at localhost/40167] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 17:01:02,817 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:01:02,817 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:01:02,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:01:02,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:01:02,818 DEBUG [Listener at localhost/40167] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13600ff0 to 127.0.0.1:54083 2023-02-13 17:01:02,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:01:02,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:01:02,818 DEBUG [Listener at localhost/40167] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:01:02,819 INFO [Listener at localhost/40167] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43327,1676307627670' ***** 2023-02-13 17:01:02,819 INFO [Listener at localhost/40167] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:01:02,819 INFO [Listener at localhost/40167] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43065,1676307627716' ***** 2023-02-13 17:01:02,819 INFO [Listener at localhost/40167] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:01:02,819 INFO [Listener at localhost/40167] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34035,1676307627747' ***** 2023-02-13 17:01:02,819 INFO [Listener at localhost/40167] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:01:02,819 INFO [RS:0;jenkins-hbase4:43327] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:01:02,819 INFO [RS:2;jenkins-hbase4:34035] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:01:02,819 INFO [RS:1;jenkins-hbase4:43065] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:01:02,819 INFO [RS:0;jenkins-hbase4:43327] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:01:02,819 INFO [RS:1;jenkins-hbase4:43065] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:01:02,819 INFO [RS:0;jenkins-hbase4:43327] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:01:02,819 INFO [RS:2;jenkins-hbase4:34035] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:01:02,820 INFO [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:01:02,820 INFO [RS:2;jenkins-hbase4:34035] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:01:02,819 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:01:02,820 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(3304): Received CLOSE for 0c94fa9feea4731bb9272766d7d4c199 2023-02-13 17:01:02,820 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:01:02,820 DEBUG [RS:2;jenkins-hbase4:34035] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77c0cd29 to 127.0.0.1:54083 2023-02-13 17:01:02,820 DEBUG [RS:2;jenkins-hbase4:34035] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:01:02,820 INFO [RS:2;jenkins-hbase4:34035] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:01:02,820 INFO [RS:2;jenkins-hbase4:34035] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:01:02,820 INFO [RS:2;jenkins-hbase4:34035] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:01:02,820 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 17:01:02,820 INFO [RS:1;jenkins-hbase4:43065] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:01:02,821 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 17:01:02,819 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:01:02,819 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:01:02,821 DEBUG [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1479): Online Regions={0c94fa9feea4731bb9272766d7d4c199=hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199., 1588230740=hbase:meta,,1.1588230740} 2023-02-13 17:01:02,821 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 17:01:02,821 INFO [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:01:02,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0c94fa9feea4731bb9272766d7d4c199, disabling compactions & flushes 2023-02-13 17:01:02,821 DEBUG [RS:1;jenkins-hbase4:43065] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x450b6030 to 127.0.0.1:54083 2023-02-13 17:01:02,820 DEBUG [RS:0;jenkins-hbase4:43327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e5196ac to 127.0.0.1:54083 2023-02-13 17:01:02,821 DEBUG [RS:1;jenkins-hbase4:43065] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:01:02,821 INFO [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43065,1676307627716; all regions closed. 2023-02-13 17:01:02,821 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:01:02,821 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 17:01:02,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:01:02,821 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 17:01:02,821 DEBUG [RS:0;jenkins-hbase4:43327] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:01:02,821 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 17:01:02,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. after waiting 0 ms 2023-02-13 17:01:02,822 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 17:01:02,822 INFO [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43327,1676307627670; all regions closed. 2023-02-13 17:01:02,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:01:02,822 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-13 17:01:02,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 0c94fa9feea4731bb9272766d7d4c199 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 17:01:02,822 DEBUG [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1505): Waiting on 0c94fa9feea4731bb9272766d7d4c199, 1588230740 2023-02-13 17:01:02,835 DEBUG [RS:0;jenkins-hbase4:43327] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/oldWALs 2023-02-13 17:01:02,835 INFO [RS:0;jenkins-hbase4:43327] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43327%2C1676307627670:(num 1676307629223) 2023-02-13 17:01:02,835 DEBUG [RS:0;jenkins-hbase4:43327] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:01:02,835 INFO [RS:0;jenkins-hbase4:43327] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:01:02,837 DEBUG [RS:1;jenkins-hbase4:43065] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/oldWALs 2023-02-13 17:01:02,837 INFO [RS:1;jenkins-hbase4:43065] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43065%2C1676307627716:(num 1676307629223) 2023-02-13 17:01:02,837 DEBUG [RS:1;jenkins-hbase4:43065] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:01:02,837 INFO [RS:1;jenkins-hbase4:43065] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:01:02,843 INFO [RS:0;jenkins-hbase4:43327] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 17:01:02,843 INFO [RS:0;jenkins-hbase4:43327] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:01:02,843 INFO [RS:1;jenkins-hbase4:43065] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 17:01:02,843 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:01:02,844 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:01:02,843 INFO [RS:0;jenkins-hbase4:43327] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:01:02,845 INFO [RS:0;jenkins-hbase4:43327] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:01:02,844 INFO [RS:1;jenkins-hbase4:43065] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:01:02,845 INFO [RS:1;jenkins-hbase4:43065] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:01:02,845 INFO [RS:1;jenkins-hbase4:43065] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:01:02,846 INFO [RS:0;jenkins-hbase4:43327] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43327 2023-02-13 17:01:02,846 INFO [RS:1;jenkins-hbase4:43065] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43065 2023-02-13 17:01:02,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199/.tmp/info/99cef9a466234f2b9f847e2c50c506a1 2023-02-13 17:01:02,851 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/.tmp/info/a622e40cd2474fee9c09de36d84bd113 2023-02-13 17:01:02,856 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:01:02,856 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:01:02,856 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:01:02,856 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43065,1676307627716 2023-02-13 17:01:02,856 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@59a6e3d5 rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:02,856 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:01:02,857 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:01:02,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199/.tmp/info/99cef9a466234f2b9f847e2c50c506a1 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199/info/99cef9a466234f2b9f847e2c50c506a1 2023-02-13 17:01:02,856 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:01:02,857 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@471f7eeb rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:02,859 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:01:02,859 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f275c32 rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:02,859 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:01:02,859 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4ff14bf0 rejected from java.util.concurrent.ThreadPoolExecutor@4bcec6d6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:02,859 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43065,1676307627716] 2023-02-13 17:01:02,859 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a622e40cd2474fee9c09de36d84bd113 2023-02-13 17:01:02,860 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43065,1676307627716; numProcessing=1 2023-02-13 17:01:02,859 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43327,1676307627670 2023-02-13 17:01:02,862 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43065,1676307627716 already deleted, retry=false 2023-02-13 17:01:02,862 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,43065,1676307627716 expired; onlineServers=2 2023-02-13 17:01:02,862 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43327,1676307627670] 2023-02-13 17:01:02,862 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43327,1676307627670; numProcessing=2 2023-02-13 17:01:02,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199/info/99cef9a466234f2b9f847e2c50c506a1, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 17:01:02,863 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43327,1676307627670 already deleted, retry=false 2023-02-13 17:01:02,863 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,43327,1676307627670 expired; onlineServers=1 2023-02-13 17:01:02,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 0c94fa9feea4731bb9272766d7d4c199 in 41ms, sequenceid=6, compaction requested=false 2023-02-13 17:01:02,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/namespace/0c94fa9feea4731bb9272766d7d4c199/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 17:01:02,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:01:02,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0c94fa9feea4731bb9272766d7d4c199: Waiting for close lock at 1676307662820Running coprocessor pre-close hooks at 1676307662820Disabling compacts and flushes for region at 1676307662820Disabling writes for close at 1676307662822 (+2 ms)Obtaining lock to block concurrent updates at 1676307662822Preparing flush snapshotting stores in 0c94fa9feea4731bb9272766d7d4c199 at 1676307662822Finished memstore snapshotting hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676307662822Flushing stores of hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. at 1676307662823 (+1 ms)Flushing 0c94fa9feea4731bb9272766d7d4c199/info: creating writer at 1676307662823Flushing 0c94fa9feea4731bb9272766d7d4c199/info: appending metadata at 1676307662828 (+5 ms)Flushing 0c94fa9feea4731bb9272766d7d4c199/info: closing flushed file at 1676307662828Flushing 0c94fa9feea4731bb9272766d7d4c199/info: reopening flushed file at 1676307662859 (+31 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 0c94fa9feea4731bb9272766d7d4c199 in 41ms, sequenceid=6, compaction requested=false at 1676307662864 (+5 ms)Writing region close event to WAL at 1676307662865 (+1 ms)Running coprocessor post-close hooks at 1676307662869 (+4 ms)Closed at 1676307662870 (+1 ms) 2023-02-13 17:01:02,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676307629637.0c94fa9feea4731bb9272766d7d4c199. 2023-02-13 17:01:02,874 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/.tmp/rep_barrier/417878f0671042008b46df3f00899427 2023-02-13 17:01:02,877 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 417878f0671042008b46df3f00899427 2023-02-13 17:01:02,885 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/.tmp/table/db5270ea9f754fe3923451bfaaa5a0ec 2023-02-13 17:01:02,887 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for db5270ea9f754fe3923451bfaaa5a0ec 2023-02-13 17:01:02,887 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/.tmp/info/a622e40cd2474fee9c09de36d84bd113 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/info/a622e40cd2474fee9c09de36d84bd113 2023-02-13 17:01:02,890 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a622e40cd2474fee9c09de36d84bd113 2023-02-13 17:01:02,890 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/info/a622e40cd2474fee9c09de36d84bd113, entries=64, sequenceid=307, filesize=11.8 K 2023-02-13 17:01:02,891 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/.tmp/rep_barrier/417878f0671042008b46df3f00899427 as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/rep_barrier/417878f0671042008b46df3f00899427 2023-02-13 17:01:02,893 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 417878f0671042008b46df3f00899427 2023-02-13 17:01:02,893 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/rep_barrier/417878f0671042008b46df3f00899427, entries=18, sequenceid=307, filesize=7.0 K 2023-02-13 17:01:02,894 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/.tmp/table/db5270ea9f754fe3923451bfaaa5a0ec as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/table/db5270ea9f754fe3923451bfaaa5a0ec 2023-02-13 17:01:02,897 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for db5270ea9f754fe3923451bfaaa5a0ec 2023-02-13 17:01:02,897 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/table/db5270ea9f754fe3923451bfaaa5a0ec, entries=22, sequenceid=307, filesize=7.0 K 2023-02-13 17:01:02,898 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 75ms, sequenceid=307, compaction requested=false 2023-02-13 17:01:02,901 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:01:02,901 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:01:02,901 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:01:02,901 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-13 17:01:02,902 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 17:01:02,903 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 17:01:02,903 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676307662821Running coprocessor pre-close hooks at 1676307662821Disabling compacts and flushes for region at 1676307662821Disabling writes for close at 1676307662822 (+1 ms)Obtaining lock to block concurrent updates at 1676307662822Preparing flush snapshotting stores in 1588230740 at 1676307662822Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141010, getHeapSize=217528, getOffHeapSize=0, getCellsCount=997 at 1676307662822Flushing stores of hbase:meta,,1.1588230740 at 1676307662823 (+1 ms)Flushing 1588230740/info: creating writer at 1676307662823Flushing 1588230740/info: appending metadata at 1676307662831 (+8 ms)Flushing 1588230740/info: closing flushed file at 1676307662831Flushing 1588230740/rep_barrier: creating writer at 1676307662860 (+29 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676307662865 (+5 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676307662865Flushing 1588230740/table: creating writer at 1676307662877 (+12 ms)Flushing 1588230740/table: appending metadata at 1676307662879 (+2 ms)Flushing 1588230740/table: closing flushed file at 1676307662879Flushing 1588230740/info: reopening flushed file at 1676307662888 (+9 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676307662891 (+3 ms)Flushing 1588230740/table: reopening flushed file at 1676307662894 (+3 ms)Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 75ms, sequenceid=307, compaction requested=false at 1676307662898 (+4 ms)Writing region close event to WAL at 1676307662899 (+1 ms)Running coprocessor post-close hooks at 1676307662902 (+3 ms)Closed at 1676307662903 (+1 ms) 2023-02-13 17:01:02,903 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 17:01:03,016 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:01:03,016 INFO [RS:0;jenkins-hbase4:43327] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43327,1676307627670; zookeeper connection closed. 2023-02-13 17:01:03,017 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7ae38629 rejected from java.util.concurrent.ThreadPoolExecutor@4bcec6d6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:03,017 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43327-0x1004ea060850001, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:01:03,017 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@24d751f rejected from java.util.concurrent.ThreadPoolExecutor@4bcec6d6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:03,017 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@ce0f3f9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@ce0f3f9 2023-02-13 17:01:03,023 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34035,1676307627747; all regions closed. 2023-02-13 17:01:03,028 DEBUG [RS:2;jenkins-hbase4:34035] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/oldWALs 2023-02-13 17:01:03,028 INFO [RS:2;jenkins-hbase4:34035] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34035%2C1676307627747.meta:.meta(num 1676307629369) 2023-02-13 17:01:03,032 DEBUG [RS:2;jenkins-hbase4:34035] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/oldWALs 2023-02-13 17:01:03,033 INFO [RS:2;jenkins-hbase4:34035] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34035%2C1676307627747:(num 1676307629223) 2023-02-13 17:01:03,033 DEBUG [RS:2;jenkins-hbase4:34035] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:01:03,033 INFO [RS:2;jenkins-hbase4:34035] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:01:03,033 INFO [RS:2;jenkins-hbase4:34035] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 17:01:03,033 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:01:03,034 INFO [RS:2;jenkins-hbase4:34035] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34035 2023-02-13 17:01:03,035 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34035,1676307627747 2023-02-13 17:01:03,035 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:01:03,035 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1258ccec rejected from java.util.concurrent.ThreadPoolExecutor@16675a7f[Terminated, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:03,037 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34035,1676307627747] 2023-02-13 17:01:03,037 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34035,1676307627747; numProcessing=3 2023-02-13 17:01:03,038 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34035,1676307627747 already deleted, retry=false 2023-02-13 17:01:03,038 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34035,1676307627747 expired; onlineServers=0 2023-02-13 17:01:03,038 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40255,1676307626519' ***** 2023-02-13 17:01:03,038 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 17:01:03,039 DEBUG [M:0;jenkins-hbase4:40255] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1956d169, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:01:03,039 INFO [M:0;jenkins-hbase4:40255] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40255,1676307626519 2023-02-13 17:01:03,039 INFO [M:0;jenkins-hbase4:40255] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40255,1676307626519; all regions closed. 2023-02-13 17:01:03,039 DEBUG [M:0;jenkins-hbase4:40255] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:01:03,039 DEBUG [M:0;jenkins-hbase4:40255] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 17:01:03,039 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 17:01:03,039 DEBUG [M:0;jenkins-hbase4:40255] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 17:01:03,039 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307628867] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307628867,5,FailOnTimeoutGroup] 2023-02-13 17:01:03,039 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307628866] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307628866,5,FailOnTimeoutGroup] 2023-02-13 17:01:03,040 INFO [M:0;jenkins-hbase4:40255] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 17:01:03,040 INFO [M:0;jenkins-hbase4:40255] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 17:01:03,040 INFO [M:0;jenkins-hbase4:40255] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 17:01:03,040 DEBUG [M:0;jenkins-hbase4:40255] master.HMaster(1502): Stopping service threads 2023-02-13 17:01:03,041 INFO [M:0;jenkins-hbase4:40255] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 17:01:03,041 ERROR [M:0;jenkins-hbase4:40255] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-13 17:01:03,041 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:01:03,041 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:01:03,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:01:03,042 INFO [M:0;jenkins-hbase4:40255] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 17:01:03,042 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 17:01:03,042 DEBUG [M:0;jenkins-hbase4:40255] zookeeper.ZKUtil(398): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 17:01:03,042 WARN [M:0;jenkins-hbase4:40255] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 17:01:03,042 INFO [M:0;jenkins-hbase4:40255] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 17:01:03,042 INFO [M:0;jenkins-hbase4:40255] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 17:01:03,042 DEBUG [M:0;jenkins-hbase4:40255] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 17:01:03,042 INFO [M:0;jenkins-hbase4:40255] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:01:03,042 DEBUG [M:0;jenkins-hbase4:40255] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:01:03,043 DEBUG [M:0;jenkins-hbase4:40255] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 17:01:03,043 DEBUG [M:0;jenkins-hbase4:40255] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:01:03,043 INFO [M:0;jenkins-hbase4:40255] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=844.86 KB heapSize=1.01 MB 2023-02-13 17:01:03,057 INFO [M:0;jenkins-hbase4:40255] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=844.86 KB at sequenceid=2097 (bloomFilter=true), to=hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/3a08ceee24f14f589ff4ada9e7ac6ecd 2023-02-13 17:01:03,060 INFO [M:0;jenkins-hbase4:40255] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3a08ceee24f14f589ff4ada9e7ac6ecd 2023-02-13 17:01:03,060 DEBUG [M:0;jenkins-hbase4:40255] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/3a08ceee24f14f589ff4ada9e7ac6ecd as hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/3a08ceee24f14f589ff4ada9e7ac6ecd 2023-02-13 17:01:03,063 INFO [M:0;jenkins-hbase4:40255] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3a08ceee24f14f589ff4ada9e7ac6ecd 2023-02-13 17:01:03,063 INFO [M:0;jenkins-hbase4:40255] regionserver.HStore(1080): Added hdfs://localhost:45307/user/jenkins/test-data/d1b08e66-b99a-73e4-dd06-2e8113dce5e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/3a08ceee24f14f589ff4ada9e7ac6ecd, entries=282, sequenceid=2097, filesize=30.6 K 2023-02-13 17:01:03,064 INFO [M:0;jenkins-hbase4:40255] regionserver.HRegion(2947): Finished flush of dataSize ~844.86 KB/865135, heapSize ~1.01 MB/1060288, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 21ms, sequenceid=2097, compaction requested=false 2023-02-13 17:01:03,064 INFO [M:0;jenkins-hbase4:40255] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:01:03,064 DEBUG [M:0;jenkins-hbase4:40255] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676307663042Disabling compacts and flushes for region at 1676307663042Disabling writes for close at 1676307663043 (+1 ms)Obtaining lock to block concurrent updates at 1676307663043Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676307663043Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=865135, getHeapSize=1060288, getOffHeapSize=0, getCellsCount=2584 at 1676307663043Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676307663043Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676307663043Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676307663048 (+5 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676307663048Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676307663061 (+13 ms)Finished flush of dataSize ~844.86 KB/865135, heapSize ~1.01 MB/1060288, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 21ms, sequenceid=2097, compaction requested=false at 1676307663064 (+3 ms)Writing region close event to WAL at 1676307663064Closed at 1676307663064 2023-02-13 17:01:03,068 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:01:03,069 INFO [M:0;jenkins-hbase4:40255] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 17:01:03,069 INFO [M:0;jenkins-hbase4:40255] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40255 2023-02-13 17:01:03,071 DEBUG [M:0;jenkins-hbase4:40255] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,40255,1676307626519 already deleted, retry=false 2023-02-13 17:01:03,117 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:01:03,117 INFO [RS:1;jenkins-hbase4:43065] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43065,1676307627716; zookeeper connection closed. 2023-02-13 17:01:03,117 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4460d77a rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:03,117 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:43065-0x1004ea060850002, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:01:03,117 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@24231187 rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:03,117 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@34593d8a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@34593d8a 2023-02-13 17:01:03,618 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:01:03,618 INFO [M:0;jenkins-hbase4:40255] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40255,1676307626519; zookeeper connection closed. 2023-02-13 17:01:03,618 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@448ec791 rejected from java.util.concurrent.ThreadPoolExecutor@6b73b90d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:03,618 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): master:40255-0x1004ea060850000, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:01:03,618 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@343a90d0 rejected from java.util.concurrent.ThreadPoolExecutor@6b73b90d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:03,718 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:01:03,718 INFO [RS:2;jenkins-hbase4:34035] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34035,1676307627747; zookeeper connection closed. 2023-02-13 17:01:03,718 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@369782fd rejected from java.util.concurrent.ThreadPoolExecutor@16675a7f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:03,718 DEBUG [Listener at localhost/40167-EventThread] zookeeper.ZKWatcher(600): regionserver:34035-0x1004ea060850003, quorum=127.0.0.1:54083, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:01:03,718 ERROR [Listener at localhost/40167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4ee40ded rejected from java.util.concurrent.ThreadPoolExecutor@16675a7f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:01:03,718 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5cc11f10] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5cc11f10 2023-02-13 17:01:03,718 INFO [Listener at localhost/40167] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-13 17:01:03,719 WARN [Listener at localhost/40167] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 17:01:03,722 INFO [Listener at localhost/40167] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:01:03,827 WARN [BP-660518971-172.31.14.131-1676307623443 heartbeating to localhost/127.0.0.1:45307] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 17:01:03,827 WARN [BP-660518971-172.31.14.131-1676307623443 heartbeating to localhost/127.0.0.1:45307] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-660518971-172.31.14.131-1676307623443 (Datanode Uuid 860591af-1224-4ad4-8461-65f27be6ecda) service to localhost/127.0.0.1:45307 2023-02-13 17:01:03,830 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data5/current/BP-660518971-172.31.14.131-1676307623443] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:01:03,830 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data6/current/BP-660518971-172.31.14.131-1676307623443] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:01:03,831 WARN [Listener at localhost/40167] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 17:01:03,835 INFO [Listener at localhost/40167] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:01:03,940 WARN [BP-660518971-172.31.14.131-1676307623443 heartbeating to localhost/127.0.0.1:45307] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 17:01:03,940 WARN [BP-660518971-172.31.14.131-1676307623443 heartbeating to localhost/127.0.0.1:45307] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-660518971-172.31.14.131-1676307623443 (Datanode Uuid 47b434d1-9a5d-47bc-ac6d-6e8ad92f2cf9) service to localhost/127.0.0.1:45307 2023-02-13 17:01:03,941 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data3/current/BP-660518971-172.31.14.131-1676307623443] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:01:03,941 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data4/current/BP-660518971-172.31.14.131-1676307623443] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:01:03,942 WARN [Listener at localhost/40167] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 17:01:03,944 INFO [Listener at localhost/40167] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:01:04,048 WARN [BP-660518971-172.31.14.131-1676307623443 heartbeating to localhost/127.0.0.1:45307] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 17:01:04,048 WARN [BP-660518971-172.31.14.131-1676307623443 heartbeating to localhost/127.0.0.1:45307] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-660518971-172.31.14.131-1676307623443 (Datanode Uuid 2b4e38c4-5192-482e-a428-68b79fca1402) service to localhost/127.0.0.1:45307 2023-02-13 17:01:04,049 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data1/current/BP-660518971-172.31.14.131-1676307623443] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:01:04,049 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/419181d1-76f5-9a31-23a1-4f6c482cbaaa/cluster_450323b5-0a0a-90ae-03cc-eb9ee8b274e4/dfs/data/data2/current/BP-660518971-172.31.14.131-1676307623443] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:01:04,077 INFO [Listener at localhost/40167] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:01:04,207 INFO [Listener at localhost/40167] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 17:01:04,293 INFO [Listener at localhost/40167] hbase.HBaseTestingUtility(1293): Minicluster is down